Stale-context detection
Determine whether stale context is detected and handled before affecting a task
When agents continue using information after it becomes outdated, this area becomes relevant. For the knowledge platform and agent product owner, the concern is specific: a cached policy remains authoritative after its source is replaced. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Define freshness according to the use, not a universal age threshold. Some historical information remains valid for a historical question, while a recent document may already be superseded by an authoritative correction.
Connect the operating question to inspectable evidence
- Knowledge platform and agent product owner
Owns the bounded decision and the consequence of acting.
- Source version history
Defines the starting context for the proposed work.
- Freshness signals
Exposes the first material uncertainty to examine.
- Freshness decision rules
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether stale context is detected and handled before affecting a task
Make the decision explicit
- Knowledge platform and agent product owner
- Agents continue using information after it becomes outdated.
- Decision
- Whether stale context is detected and handled before affecting a task.
- Freshness signals
- Use source version history to establish the relevant starting condition.
For the knowledge platform and agent product owner, this becomes relevant when agents continue using information after it becomes outdated. The immediate decision is whether stale context is detected and handled before affecting a task. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a cached policy remains authoritative after its source is replaced. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine freshness signals alongside invalidation behaviour
Keep the review distinctions visible
- Freshness signals
- Identify the version and age signal available at the point of use.
- Invalidation behaviour
- Update a source and trace whether cached context is invalidated.
- Uncertainty when currency is unknown
- Require an explicit uncertainty path when currentness cannot be established.
Freshness signals
For this scope, freshness signals should be examined alongside source version history. Define freshness using the source’s role in the task. An old historical record may be appropriate, while a recently cached instruction may already be superseded. The proposed cases should establish authoritative replacement events and expected answers so staleness is judged against meaning rather than age alone.
Invalidation behaviour
For this scope, invalidation behaviour should be examined alongside refresh and cache rules. Trace an update through source storage, index refresh, caches, and context assembly where those paths are in scope. Repeat a relevant task after each meaningful transition. This identifies where superseded information remains influential and which owner can address the propagation gap.
Uncertainty when currency is unknown
For this scope, uncertainty when currency is unknown should be examined alongside known stale fixtures. Create a case where the deployment cannot establish which version governs. Define whether it should qualify the answer, retrieve additional evidence, or escalate. The evaluation should not reward a confident current-sounding answer when the system lacks a reliable basis for claiming the information is current.
Prerequisites
Start with source version history
Give each input a specific job
- Source version history
- Defines the operating boundary.
- Refresh and cache rules
- Examine invalidation behaviour.
- Known stale fixtures
- Shows conditions affecting uncertainty when currency is unknown.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is source version history, refresh and cache rules, and known stale fixtures. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Freshness signals
Use source version history to define the decision criteria.
- Invalidation behaviour
Inspect refresh and cache rules and retain observations that challenge the expected result.
- Uncertainty when currency is unknown
Use known stale fixtures to identify unresolved conditions and a responsible owner.
- Freshness decision rules
Link the recommendation to its evidence, remaining limits and next validation condition.
Create stale fixtures with known replacement events and authoritative versions. Test how source updates reach indexes, caches, assembled prompts, and ongoing sessions within the agreed boundary. The proposed evaluation should also include a case where currency cannot be established and define the expected qualification or escalation behaviour.
A practical first slice would focus on freshness signals and then follow its consequences into invalidation behaviour. Keep uncertainty when currency is unknown visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Superseded cached process: propagation finding
Examine the constructed decision boundary
Illustrative scenario
- Superseded cached process
An assistant answers from an obsolete document retained in retrieval cache.
- Source replacement
Update the document and repeat the same question through the normal path.
- Propagation finding
Locate the delay and decide whether refresh, version metadata or answer qualification needs repair.
Consider an illustrative situation in which an internal assistant answers a process question using a superseded document held in a retrieval cache. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the superseded process document, an illustrative run would replace the source and repeat the question through the normal deployment path. The resulting trace would locate any delayed propagation. The owner could then decide whether refresh behaviour, version metadata, or answer qualification needs improvement for that use.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves uncertainty when currency is unknown unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use freshness decision rules to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Staleness evaluation cases
- Boundary — source version history.
- Refresh-path findings
- Findings about invalidation behaviour.
- Freshness decision rules
- Action on uncertainty when currency is unknown.
Indicative outputs include staleness evaluation cases, refresh-path findings, and freshness decision rules. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include staleness evaluation cases.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from refresh-path findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.