Cross-user memory leakage testing
Determine whether identity boundaries hold across writing, retrieval, and reuse
When a memory-enabled agent serves several users or tenants, this area becomes relevant. For the platform security and privacy owner, the concern is specific: information written for one identity influences another identity's answer. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Use approved synthetic identities and marker data to test both direct retrieval and indirect influence. A clean answer alone cannot explain whether an unauthorised entry was retrieved, suppressed later, or never encountered by the test.
Connect the operating question to inspectable evidence
- Platform security and privacy owner
Owns the bounded decision and the consequence of acting.
- Synthetic user and tenant fixtures
Defines the starting context for the proposed work.
- Cross-user queries
Exposes the first material uncertainty to examine.
- Boundary remediation findings
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether identity boundaries hold across writing, retrieval, and reuse
Make the decision explicit
- Platform security and privacy owner
- A memory-enabled agent serves several users or tenants.
- Decision
- Whether identity boundaries hold across writing, retrieval, and reuse.
- Cross-user queries
- Use synthetic user and tenant fixtures to establish the relevant starting condition.
For the platform security and privacy owner, this becomes relevant when a memory-enabled agent serves several users or tenants. The immediate decision is whether identity boundaries hold across writing, retrieval, and reuse. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that information written for one identity influences another identity’s answer. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine cross-user queries alongside tenant switching
Keep the review distinctions visible
- Cross-user queries
- Use separate synthetic principals for allowed and denied memory requests.
- Tenant switching
- Change the active tenant while retaining the surrounding session conditions.
- Shared cache and summary boundaries
- Inspect cache and summary paths that may outlive the identity switch.
Cross-user queries
For this scope, cross-user queries should be examined alongside synthetic user and tenant fixtures. Use paired synthetic users with different authorised facts but similar requests. Inspect both direct retrieval and indirect answer influence. This prevents a test from relying solely on whether the agent repeats an obvious marker and helps identify less visible mixing of one user’s context into another’s task.
Tenant switching
For this scope, tenant switching should be examined alongside authorisation model. Exercise tenant changes through the deployment’s actual session and identity mechanisms. Reusing a browser session, conversation, or worker process may create different test conditions from a completely fresh run. Preserve the acting tenant and permitted data scope with every trace so the boundary can be reviewed.
Shared cache and summary boundaries
For this scope, shared cache and summary boundaries should be examined alongside memory access traces. Identify shared caches, summaries, and derived stores before testing. Place distinct synthetic markers in the relevant paths and inspect their reuse. An entry that is filtered from the primary store may still be influential elsewhere, so the result should name which representations were actually examined.
Prerequisites
Start with synthetic user and tenant fixtures
Give each input a specific job
- Synthetic user and tenant fixtures
- Defines the operating boundary.
- Authorisation model
- Examine tenant switching.
- Memory access traces
- Shows conditions affecting shared cache and summary boundaries.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is synthetic user and tenant fixtures, authorisation model, and memory access traces. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Cross-user queries
Use synthetic user and tenant fixtures to define the decision criteria.
- Tenant switching
Inspect authorisation model and retain observations that challenge the expected result.
- Shared cache and summary boundaries
Use memory access traces to identify unresolved conditions and a responsible owner.
- Boundary remediation findings
Link the recommendation to its evidence, remaining limits and next validation condition.
Construct paired synthetic identities whose requests are deliberately similar but whose permitted facts differ. Exercise fresh sessions, reused sessions, role changes, and any shared cache paths in scope. The proposed test should retain identity-linked retrieval evidence so reviewers can distinguish a true boundary failure from coincidental generic output.
A practical first slice would focus on cross-user queries and then follow its consequences into tenant switching. Keep shared cache and summary boundaries visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Switched synthetic tenant: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Switched synthetic tenant
A session changes principal while shared cache or summary state remains.
- Isolation challenge
Request comparable memories from each test identity and inspect access-linked traces.
- Boundary finding
Record the exact retained-state path before proposing remediation.
Consider an illustrative situation in which two synthetic customer accounts ask similar questions after only one account has supplied a private preference. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the two-account example, a marker supplied only to one account would be queried directly and indirectly from the other. An illustrative finding would record where the marker entered, which identity retrieved it, and how it influenced the answer. Unobserved stores or provider paths should remain explicit limitations.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves shared cache and summary boundaries unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use boundary remediation findings to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Isolation test matrix
- Boundary — synthetic user and tenant fixtures.
- Identity-linked traces
- Findings about tenant switching.
- Boundary remediation findings
- Action on shared cache and summary boundaries.
Indicative outputs include isolation test matrix, identity-linked traces, and boundary remediation findings. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include isolation test matrix.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from identity-linked traces.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.