Comparative sandbox benchmarking
Compare sandbox options against the same workload and control requirements
Several execution environments are being considered for one agent deployment. The decision is how to compare sandbox options against the same workload and control requirements. The work begins with your actual workflow and looks for the evidence that separates intended behaviour from a consequential exception. It is particularly relevant to platform buyers and security architecture teams. A useful outcome is an explicit account of what was examined, what remains uncertain, and which decision the evidence can support.
The runtime isolation relationship
- Candidate configurations
Establish the basis for common workload.
- Required isolation
Compare candidates against the same boundary needs.
- Comparative benchmark specification
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around common workload
Questions and the evidence they need
- Common workload
- Inspect candidate configurations for the relevant constraint or observation.
- Required isolation
- Inspect common workload fixtures for the relevant constraint or observation.
- Operating constraints
- Inspect isolation criteria for the relevant constraint or observation.
The decision is how to compare sandbox options against the same workload and control requirements. This is relevant to platform buyers and security architecture teams when several execution environments are being considered for one agent deployment. Name the affected workflow, the consequence of a wrong decision, and the owner who can act on the result. The initial discussion should establish which evidence is already available and which uncertainty is material enough to investigate.
The specific failure hypothesis is that a fast default configuration is compared with a more restricted configuration as if their controls were equal. It remains a hypothesis until examined in the deployment. Existing internal tests or operating records may already resolve it; identify the uncovered question before commissioning additional work.
Mechanism
Sandbox comparisons need equivalent workload and controls
Relationships within runtime isolation
- Common workload
- Hold the tested task and fixture conditions visible.
- Required isolation
- Compare candidates against the same boundary needs.
- Operating constraints
- Expose cleanup, recovery, and cost-input differences.
What this specialty examines
Comparative sandbox benchmarking needs equivalent declared requirements. List isolation, egress, storage, cleanup and operational settings alongside performance measurements. A candidate configured with fewer restrictions should not appear unqualifiedly faster under a supposedly identical task.
The review should hold workload and required controls explicit, then compare observed behaviour and operational limitations. Identify the normal route, relevant exceptions and the authoritative record of the outcome. Inspect the relevant business-system record, enforcement event or process evidence alongside any reported status. Keep incomplete cases visible, with an explanation of what their absence prevents the owner from concluding.
The tradeoff to make explicit
For this decision, performance comparisons are useful only alongside equivalent scope and security expectations. Compare the available design or operating choices against the same intended outcome. If one option removes a required control or changes the task boundary, describe that difference before treating its apparent simplicity, speed or lower effort as an improvement. The decision should expose the tradeoff instead of burying it in an aggregate score.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for common workload
Illustrative working format
- Candidate configurations
- Match this description to the deployed or proposed boundary being examined.
- Common workload fixtures
- Keep workload, quality, human effort, and uncertain inputs visible in comparisons.
- Isolation criteria
- Confirm the applicable revision and who resolves disputed interpretation.
- Cost inputs
- Keep workload, quality, human effort, and uncertain inputs visible in comparisons.
Start with candidate configurations, common workload fixtures, isolation criteria and cost inputs. Identify the owner, revision and operating context of each source. Sanitised examples may support initial design, but differences from actual conditions should remain explicit. Where direct access is unsuitable, a customer-controlled evidence or execution route may be considered, with agreed handling and retention.
The scope also needs the runtime owner, agreed expectations for a representative boundary test, and the authority to inspect the relevant boundary. Specify what evidence would distinguish the failure in which a fast default configuration is compared with a more restricted configuration as if their controls were equal from acceptable behaviour. If stakeholders disagree about intended behaviour, resolve the interpretation before classifying the observed difference as a defect.
How the work proceeds
Carry common workload through to an accountable finding
From the scoped material to a decision record
- Candidate configurations
Declare the permitted environment and resources.
- Required isolation
Apply the scoped check to operating constraints.
- Comparative benchmark specification
Record the evidence, open dependencies, and the owner’s next decision.
Prepare the runtime boundary and fixtures
Use candidate configurations, common workload fixtures, isolation criteria and cost inputs to identify the exact runtime and boundary claimed by platform buyers and security architecture teams. Mark permitted resources and protected resources in a controlled fixture, including shared services where they are relevant. Define observable success, denial and incomplete setup before execution. The preparation output is a boundary map and fixture manifest that lets reviewers distinguish effective isolation from a test that never reached its target.
Exercise and inspect the boundary
Use the same representative workload and record configuration differences that cannot be normalised. Report startup, execution and recovery observations separately where useful. Include untested controls and operating dependencies in the option matrix rather than forcing a single winner.
Choose containment or configuration changes
Translate the observations into a comparative benchmark specification and evidence-backed option matrix. Locate each issue in the worker, shared service, policy rollout or cleanup path rather than attributing every symptom to the sandbox as a whole. Decide which configuration change, narrower resource exposure or specialist investigation is warranted. Retest the affected boundary and inspect residual state before treating the changed environment as equivalent to the intended design.
Indicative artefact
Make the comparative benchmark specification usable by its owner
Comparative benchmark specification
Illustrative working format
- Comparative benchmark specification
- Identify the assessed workflow, revision, and decision boundary.
- Evidence-backed option matrix
- Link observations and counterevidence to the case, configuration, and conclusion limit.
- Required isolation evidence
- Link observations and counterevidence to the case, configuration, and conclusion limit.
A useful decision artefact is a comparative benchmark specification and evidence-backed option matrix. Supporting material may include selected cases, configuration or process records, evidence references, a disagreement log and prioritised actions. Retain the case selection rationale and material disagreements alongside the result; otherwise the result may be mistaken for complete coverage.
Illustrative scenario
Examine operating constraints through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Common workload
One candidate appears simpler only because a required restriction is absent.
- Required isolation
Align required restrictions across the candidate configurations.
- Operating constraints
Reject a convenience comparison that silently omits a required control.
Consider a fictional situation: two coding runtimes are compared on startup time even though only one configuration has restricted egress. The scenario is illustrative, with no client engagement, executed test or measured improvement implied.
For the startup comparison, establish whether both candidates satisfy the required egress boundary. If only one does, the result is a tradeoff between different configurations. The decision owner may need a new configuration test before selecting an environment.
Scope and next action
Keep the conclusion within the authorised runtime boundary
Confirm fit before the next commitment
- Common workload
- This review is limited to the authorised runtime boundary.
- Candidate configurations
- The relevant owner must confirm access and the intended operating conditions.
- Comparative benchmark specification
- Untested surfaces remain outside the conclusion.
Testing is confined to explicitly authorised environments and non-destructive cases. A bounded assessment cannot establish that escape or cross-boundary access is impossible. Any engagement depends on an agreed scope, qualified team, authorised access and confirmed availability. Bring the affected workflow, its current configuration or process, the triggering change and the evidence already available. The immediate scope question is prompted by the situation in which several execution environments are being considered for one agent deployment; confirm which missing observation would change the owner’s next action.
Ways to engage
Related engagement scopes
AI Agent Runtime Security Review
Make every delegation explicit. Examine who the agent acts for, where it can execute and which tools it can reach when instructions alone are not enough.
AI Agent Red Team
Challenge the boundaries that matter to your workflow. Test agreed attack scenarios against agent instructions, tools and data access.
AI Agent Sandbox Assurance
Examine the boundary between agent execution and the systems around it, including files, networks, secrets and cleanup.
Continue exploring
Related insights and resources
The permission boundary behind every agent action
An agent can only be evaluated in the context of what its tools allow. Examine identity, delegated authority and action boundaries before you put a workflow in motion.
Before your agent can act, define its authority
A working paper for the moment an AI system moves from advice to action. Map identity, permissions, approvals and recovery into an authority model your team can review and test.
Computer-use agents bring the interface back into the threat model
A computer-use agent treats pixels, accessibility data and interface state as both evidence and an action surface. Security must bind intent to effects beyond the screen.