Capability-growth evaluation
Check whether apparent capability growth survives controlled comparison
Capability-growth evaluation becomes relevant when an agent is reported to improve after experience, tool changes or self-generated adaptations. A focused review would examine a capability claim compared across agent states using protected tasks and declared resources. This is most relevant to AI research leads and adaptive-agent product owners, especially where teams must distinguish capability growth and extra resources.
Follow the agent through a declared change sequence
- Before-and-after agent states
Identify the initial state and what can be reconstructed.
- Capability growth
Introduce or observe the relevant changed condition.
- Held-out task and behaviour cases
Compare behaviour while retaining version and workload context.
- A capability and authority review brief
Define the evidence needed before extending the conclusion.
Decision at stake
Check whether apparent capability growth survives controlled comparison
Make the decision explicit
- AI research leads and adaptive-agent product owners
- An agent is reported to improve after experience, tool changes or self-generated adaptations.
- Decision
- Which improvements are supported, what changed and whether new capabilities require different controls.
- Capability growth
- Use before-and-after agent states to establish the relevant starting condition.
For AI research leads and adaptive-agent product owners, the immediate question is which improvements are supported, what changed and whether new capabilities require different controls. This is most useful when there is a real choice between proceeding, changing the design, collecting more evidence or narrowing the intended use. The starting conversation should identify that choice and its consequence before deciding how much work the topic warrants.
An agent could score better because it saw the test cases or gained stronger tools, while the claim attributes the change to learning. A useful review follows a capability claim compared across agent states using protected tasks and declared resources, rather than stopping at a convenient component metric or document boundary. The evidence should separate observation, interpretation and recommendation so the accountable owner can see what it supports.
Coverage
Capability growth and extra resources need different evidence
Keep the review distinctions visible
- Capability growth
- Compare with extra resources.
- Training familiarity
- Compare with transferable skill.
- Improved success
- Compare with new authority exposure.
Capability growth versus extra resources
The claim would name the capability and intended transfer beyond familiar examples. Protected tasks and documented resources would challenge whether improvement reflects learning, test exposure or additional tool assistance.
Training familiarity versus transferable skill
Before-and-after states would include relevant prompts, memory and operating conditions. A comparison that changes several components at once may still be useful, but attribution to one mechanism would remain limited.
Improved success versus new authority exposure
New actions would receive an authority review alongside performance analysis. An agent that completes more tasks may also attempt broader operations, making capability evidence and autonomy approval separate decisions.
Prerequisites
Start with before-and-after agent states
Give each input a specific job
- Before-and-after agent states
- Defines the operating boundary.
- Held-out task and behaviour cases
- Examine training familiarity.
- Tool access, resources and adaptation records
- Shows conditions affecting improved success.
- Agent operating owner
- Confirm access and a reviewer who can resolve gaps.
For Capability-growth evaluation, the team should be able to identify the relevant model, prompt, tool and retained-state conditions. A controlled environment, representative sequences and explicit stop conditions are needed for active tests. Missing lineage or inaccessible provider behaviour would be recorded as a limitation on reconstruction and comparison.
Useful starting material would include before-and-after agent states; held-out task and behaviour cases; tool access, resources and adaptation records. These inputs have different jobs: the first establishes context, the second supplies behaviour or comparison material, and the third exposes operating conditions that may qualify a conclusion. Existing material would be reviewed before requesting new collection, and a missing input would be recorded as an evidence gap rather than treated as a favourable result.
How the work proceeds
Preserve the state sequence before interpreting change
Follow the evidence through the work
- Before-and-after agent states
Fix the state, sequence and stopping conditions for the comparison.
- Held-out task and behaviour cases
Preserve this evidence with the version and operating context.
- Training familiarity
Challenge this distinction across the declared change sequence.
- A capability and authority review brief
State which change should trigger another test or operating restriction.
Define state and sequence
Define the claimed capability and plausible alternative explanations. The opening step turns the operating concern into a question that can be examined. It would identify a capability claim compared across agent states using protected tasks and declared resources, the outcome the owner cares about and the conditions that would make an observation decision-relevant. Scope changes would remain explicit if the evidence reveals a different problem.
Protect the comparison
Freeze comparison states, resources and protected evaluation cases. The comparison material would include held-out task and behaviour cases, with tool access, resources and adaptation records available to explain the operating conditions. Differences between capability growth and extra resources would remain visible beside the observation rather than disappear into a pooled result.
Observe change and failure
Measure transfer, adverse behaviour and relevant operating costs. The review question would explicitly distinguish training familiarity from transferable skill. A material discrepancy would be traced back to before-and-after agent states and the relevant execution or decision record, so a proposed change addresses the observed cause rather than the most convenient explanation.
Set the next operating decision
Report which claim is supported and what new controls or tests are needed. The resulting capability and authority review brief would explain the distinction between improved success and new authority exposure and identify what further observation would change the recommendation. The agent operating owner would retain the decision, including whether a narrower scope or a different approach is preferable.
Illustrative scenario
Learning from prior cases: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Learning from prior cases
Retained task examples coincide with apparently better planning.
- Protected comparison
Use held-out work and declared tools and resource budgets.
- Authority check
Separate supported task improvement from attempts to exceed prior permissions.
Imagine a fictional planning agent that improves after retaining previous task examples. The proposed evaluation would use held-out cases, document available tools and compare resource budgets. It would also examine whether the new behaviour attempts actions outside the earlier boundary. The decision would separate a supported task improvement from unsupported claims of general capability growth.
The improvement might disappear on tasks whose examples were never retained. The report would qualify the growth claim and identify whether the method supports useful specialization instead. That narrower result could still inform a product decision without being presented as broad or transferable capability growth. These are hypothetical decision paths, not observed client outcomes.
Indicative artefacts
Use a capability and authority review brief to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- An agent capability-growth claim protocol
- Boundary — before-and-after agent states.
- A controlled comparison evidence pack
- Findings about training familiarity.
- A capability and authority review brief
- Action on improved success.
The final artefact set depends on the agreed decision. Useful outputs can include:
- An agent capability-growth claim protocol. The working foundation for the question, including relevant definitions, boundaries and assumptions so another reviewer can understand what was examined.
- A controlled comparison evidence pack. The substantive observations and interpretation, preserving adverse results, disagreement and limitations that affect how the evidence may be used.
- A capability and authority review brief. The connection to action: what the owner could do next, what evidence is still missing and what change would justify revisiting the conclusion.
Scope and authority
Evidence from a sequence cannot establish indefinite behaviour
Keep analytical work separate from decision authority
- Versioned states and controlled sequences
- Can include an agent capability-growth claim protocol.
- Agent operating owner
- Operating owners authorise experiments, deployment changes and stop conditions.
- Indefinite reliability or alignment guarantees
- Cannot be inferred from a controlled comparison evidence pack.
These methods provide bounded behavioural evidence, not proof of indefinite reliability, alignment or beneficial self-improvement. Research simulations are not operating history. Active tests require authorised environments, competent reviewers and agreed stop conditions; the customer retains deployment and risk decisions. To discuss capability-growth evaluation, the first practical step is to identify the relevant owner and bring a sanitised description of the decision, the current alternative and the material uncertainty. That gives scoping a concrete starting point without implying a preapproved package or favourable conclusion.
Ways to engage
Related engagement scopes
AI Fractional Agentic AI Office
Create a regular place for agentic AI decisions. Connect initiative priorities, architecture questions, risk ownership and evidence reviews.
AI Fractional Agentic Transformation Office
Coordinate the operating changes behind agent adoption. Align workflow redesign, ownership, dependencies and adoption measures.
AI Harness Engineering Pod
Engineer a bounded AI execution harness with tool adapters, context handling, orchestration, recovery and observable traces.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
Governing agent decisions
Governance becomes useful when it changes a decision. Connect each agent workflow to an accountable owner, explicit operating conditions, proportionate evidence and a practical route for exceptions and withdrawal.
Do you need an agent centre of excellence?
A centre can concentrate scarce expertise and reusable controls, or become an approval queue detached from real work. Choose the mandate before the structure.