Sandbox Assurance Benchmark
Compare sandbox boundaries using declared workloads and threat assumptions
Sandbox Assurance Benchmark becomes relevant when several isolation approaches are being considered for agents that execute code or use computers. A focused review would examine a sandbox configuration under an authorised, reproducible workload and threat model. This is most relevant to platform security leaders and systems researchers, especially where teams must distinguish isolation claims and observed boundaries.
Connect the operating question to inspectable evidence
- Platform security leaders and systems researchers
- Owns the bounded decision and the consequence of acting.
- Isolation architecture and configuration
- Defines the starting context for the proposed work.
- Isolation claims versus observed boundaries
- Exposes the first material uncertainty to examine.
- An operational tradeoff and retest brief
- Carries unresolved conditions into an accountable next action.
Decision at stake
Compare sandbox boundaries using declared workloads and threat assumptions
Make the decision explicit
- Platform security leaders and systems researchers
- Several isolation approaches are being considered for agents that execute code or use computers.
- Decision
- Which sandbox configuration deserves further controlled testing for a specific execution boundary.
- Isolation claims
- Use isolation architecture and configuration to establish the relevant starting condition.
For platform security leaders and systems researchers, the immediate question is which sandbox configuration deserves further controlled testing for a specific execution boundary. This is most useful when there is a real choice between proceeding, changing the design, collecting more evidence or narrowing the intended use. The starting conversation should identify that choice and its consequence before deciding how much work the topic warrants.
A benchmark could mistake a blocked legitimate workflow for strong isolation or miss residual state after a successful reset. A useful review follows a sandbox configuration under an authorised, reproducible workload and threat model, rather than stopping at a convenient component metric or document boundary. The evidence should separate observation, interpretation and recommendation so the accountable owner can see what it supports.
Coverage
Isolation claims and observed boundaries need different evidence
Keep the review distinctions visible
- Isolation claims
- Compare with observed boundaries.
- Restricted workload
- Compare with general execution.
- Containment behaviour
- Compare with operational usability.
Isolation claims versus observed boundaries
The workload definition would specify which executables, files and network destinations legitimate work needs. That allows the comparison to identify a restriction that blocks required functionality without treating it as a failed security control.
Restricted workload versus general execution
Lifecycle cases would examine the beginning and end of a session, including temporary state and resource cleanup. A clean initial environment and a clean reset are different observations and would be recorded separately.
Containment behaviour versus operational usability
Security findings would be tied to the authorised threat model and reviewed by a competent specialist. The research comparison would avoid extrapolating from a limited set of boundary tests to universal resistance against escape.
Prerequisites
Start with isolation architecture and configuration
Give each input a specific job
- Isolation architecture and configuration
- Defines the operating boundary.
- Allowed workload and egress rules
- Examine restricted workload.
- Reset, cleanup and resource-limit evidence
- Shows conditions affecting containment behaviour.
- Research sponsor
- Confirm access and a reviewer who can resolve gaps.
For Sandbox Assurance Benchmark, a research question and permitted evidence route should be agreed before a dataset is assembled. Contributions would need provenance, inclusion rules and a record of collection conditions. Where a participating organisation supplies only summaries, the methodology should explain how that limits verification.
Useful starting material would include isolation architecture and configuration; allowed workload and egress rules; reset, cleanup and resource-limit evidence. These inputs have different jobs: the first establishes context, the second supplies behaviour or comparison material, and the third exposes operating conditions that may qualify a conclusion. Existing material would be reviewed before requesting new collection, and a missing input would be recorded as an evidence gap rather than treated as a favourable result.
How the work proceeds
Protect the comparison before interpreting the research
Follow the evidence through the work
- Isolation architecture and configuration
Define the comparison and freeze the conditions it covers.
- Allowed workload and egress rules
Collect this material under the declared inclusion and handling rules.
- Restricted workload
Challenge competing explanations before drawing a comparative conclusion.
- An operational tradeoff and retest brief
Retain reproducibility gaps and the research question that remains open.
Frame the comparison
Specify workloads, trust boundaries and authorised test categories. The opening step turns the operating concern into a question that can be examined. It would identify a sandbox configuration under an authorised, reproducible workload and threat model, the outcome the owner cares about and the conditions that would make an observation decision-relevant. Scope changes would remain explicit if the evidence reveals a different problem.
Collect comparable evidence
Freeze environment images and document differences that limit comparability. The comparison material would include allowed workload and egress rules, with reset, cleanup and resource-limit evidence available to explain the operating conditions. Differences between isolation claims and observed boundaries would remain visible beside the observation rather than disappear into a pooled result.
Challenge interpretation
Exercise permitted resource, filesystem, network and lifecycle conditions. The review question would explicitly distinguish restricted workload from general execution. A material discrepancy would be traced back to isolation architecture and configuration and the relevant execution or decision record, so a proposed change addresses the observed cause rather than the most convenient explanation.
Bound the conclusion
Review adverse findings with a qualified security specialist before interpretation. The resulting operational tradeoff and retest brief would explain the distinction between containment behaviour and operational usability and identify what further observation would change the recommendation. The research sponsor would retain the decision, including whether a narrower scope or a different approach is preferable.
Illustrative scenario
Document conversion: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Document conversion
One environment blocks a needed tool; another retains files between sessions.
- Two findings
Keep compatibility and cleanup observations separate in the comparison.
- Bounded retest
Select a workload-specific next test and route suspected escapes to specialists.
A fictional platform team is comparing execution environments for document-processing agents. One environment blocks a needed conversion tool, while another leaves temporary files available to the next session. The proposed benchmark would record both compatibility and cleanup findings. The decision would concern a specific workload and configuration, with specialist investigation for any suspected boundary escape.
The team could decide that the conversion restriction is acceptable if an approved alternative tool meets the workflow need. Residual files would require a different response: a cleanup investigation and retest. Keeping those findings separate avoids implying that operational inconvenience and isolation failure are interchangeable benchmark penalties. These are hypothetical decision paths, not observed client outcomes.
Indicative artefacts
Use an operational tradeoff and retest brief to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- A benchmark threat and workload specification
- Boundary — isolation architecture and configuration.
- Configuration-specific boundary results
- Findings about restricted workload.
- An operational tradeoff and retest brief
- Action on containment behaviour.
The final artefact set depends on the agreed decision. Useful outputs can include:
- A benchmark threat and workload specification. The working foundation for the question, including relevant definitions, boundaries and assumptions so another reviewer can understand what was examined.
- Configuration-specific boundary results. The substantive observations and interpretation, preserving adverse results, disagreement and limitations that affect how the evidence may be used.
- An operational tradeoff and retest brief. The connection to action: what the owner could do next, what evidence is still missing and what change would justify revisiting the conclusion.
Scope and authority
A research proposal does not establish an index or assurance scheme
Keep analytical work separate from decision authority
- Declared research question and protocol
- Can include a benchmark threat and workload specification.
- Research sponsor
- Research scope, contributors and publication rights require explicit agreement.
- Universal rankings or accredited assurance
- Cannot be inferred from configuration-specific boundary results.
A proposed research asset is not an established index, certification scheme or delivered client result. Publication, contributor participation and any public comparison would require separate authorisation. Reviewer competence, conflicts and the permitted conclusion would be confirmed for the actual assignment. To discuss sandbox assurance benchmark, the first practical step is to identify the relevant owner and bring a sanitised description of the decision, the current alternative and the material uncertainty. That gives scoping a concrete starting point without implying a preapproved package or favourable conclusion.