Permission-aware retrieval testing
Determine whether authorised context is selected for the acting identity
When retrieval must respect changing user or tenant access, this area becomes relevant. For the identity, search, and data security owner, the concern is specific: a previously authorised document remains retrievable after the user's access is removed. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Inspect retrieval-time enforcement as well as the final answer. A system that retrieves restricted evidence and relies only on later suppression has a different boundary from one that prevents the unauthorised retrieval itself.
Connect the operating question to inspectable evidence
- Identity, search, and data security owner
Owns the bounded decision and the consequence of acting.
- Permission fixtures
Defines the starting context for the proposed work.
- Access revocation
Exposes the first material uncertainty to examine.
- Revocation verification cases
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether authorised context is selected for the acting identity
Make the decision explicit
- Identity, search, and data security owner
- Retrieval must respect changing user or tenant access.
- Decision
- Whether authorised context is selected for the acting identity.
- Access revocation
- Use permission fixtures to establish the relevant starting condition.
For the identity, search, and data security owner, this becomes relevant when retrieval must respect changing user or tenant access. The immediate decision is whether authorised context is selected for the acting identity. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a previously authorised document remains retrievable after the user’s access is removed. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine access revocation alongside inherited permissions
Keep the review distinctions visible
- Access revocation
- Revoke access after indexing and test the retrieval path again.
- Inherited permissions
- Compare direct and inherited permissions for the same document.
- Cache and identity switching
- Switch principals while checking cached results and identity propagation.
Access revocation
For this scope, access revocation should be examined alongside permission fixtures. Change a synthetic user’s access and repeat the relevant query through the ordinary deployment path. Inspect retrieval outputs, not only the answer. This demonstrates whether revocation reached the boundary being evaluated and identifies any delay or stale representation within the authorised scope.
Inherited permissions
For this scope, inherited permissions should be examined alongside document access metadata. Exercise inherited access through the group or role mechanisms the customer actually uses. A direct user permission test may not cover those paths. Preserve the applicable identity and document metadata with each case so a reviewer can explain why a particular retrieval should have been allowed or denied.
Cache and identity switching
For this scope, cache and identity switching should be examined alongside retrieval and identity traces. Reuse and reset sessions or caches according to a predeclared matrix. A previously authorised passage may remain in assembled context after access changes. The result should distinguish permission filtering, cache invalidation, and session lifecycle behaviour so remediation targets the pathway that preserved the restricted information.
Prerequisites
Start with permission fixtures
Give each input a specific job
- Permission fixtures
- Defines the operating boundary.
- Document access metadata
- Examine inherited permissions.
- Retrieval and identity traces
- Shows conditions affecting cache and identity switching.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is permission fixtures, document access metadata, and retrieval and identity traces. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Access revocation
Use permission fixtures to define the decision criteria.
- Inherited permissions
Inspect document access metadata and retain observations that challenge the expected result.
- Cache and identity switching
Use retrieval and identity traces to identify unresolved conditions and a responsible owner.
- Revocation verification cases
Link the recommendation to its evidence, remaining limits and next validation condition.
Create permission transitions that occur between otherwise equivalent queries. Include access granted, removed, and changed through group or role membership where those paths are in scope. The proposed test should inspect retrieval outputs and caches with synthetic markers so unauthorised influence can be traced without using real private documents.
A practical first slice would focus on access revocation and then follow its consequences into inherited permissions. Keep cache and identity switching visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Revoked document access: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Revoked document access
An indexed document was available before the user's authorisation changed.
- Retrieval retest
Repeat the request after revocation while inspecting cache and acting identity.
- Permission finding
Distinguish source access removal from effective downstream exclusion.
Consider an illustrative situation in which a synthetic employee changes role and asks the same agent question before and after the permission update. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the role-changing employee, an illustrative sequence would repeat the same question before and after revocation. A finding should identify whether the access change reached the retrieval boundary and whether stale context persisted in the session. The remediation decision may concern identity propagation, filtering, cache invalidation, or session handling.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves cache and identity switching unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use revocation verification cases to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Permission-aware retrieval matrix
- Boundary — permission fixtures.
- Access-linked findings
- Findings about inherited permissions.
- Revocation verification cases
- Action on cache and identity switching.
Indicative outputs include permission-aware retrieval matrix, access-linked findings, and revocation verification cases. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include permission-aware retrieval matrix.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from access-linked findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.