How to evaluate a software-engineering agent on real repository work

A coding agent changes three lines, runs the available tests and announces success. The patch passes. It also bypasses an authorisation check that the visible tests did not cover and adds a dependency the repository policy prohibits.

“Tests passed” is evidence, but not enough evidence. Software-engineering work includes understanding an issue, locating the right change surface, preserving hidden contracts, working safely with tools and producing a patch another engineer can review and maintain. An evaluation should reproduce that job, not reward a plausible diff in isolation.

Build tasks from the work you actually accept

Public benchmarks are useful because they move evaluation beyond snippets. The original SWE-bench assembled 2,294 tasks from real GitHub issues and corresponding pull requests across 12 Python repositories, requiring systems to understand and modify repository-level code. That makes it a better reference than a code-completion quiz.

It is still not your acceptance test. Construct a private suite from representative task families:

• localized defect fixes;
• changes spanning modules or services;
• dependency and configuration updates;
• performance or concurrency problems;
• test creation for uncovered behaviour;
• documentation or migration changes tied to code; and
• security-sensitive changes involving identity, data or external calls.

Use resolved historical tasks where licensing and confidentiality permit, then remove the final patch and any leaked answer. Add newly authored tasks and recent repository snapshots to reduce contamination. Include work the agent should decline because the requirement is ambiguous or the environment lacks necessary evidence.

Define success beyond the visible test suite

For each task, write an acceptance contract before running the agent.

Dimension	Evidence
Functional	visible and hidden tests, reproduced defect, required behaviour
Regression	unaffected suites, compatibility checks, performance bounds
Security	static analysis, dependency policy, permission and data-flow review
Scope	relevant files changed, no unrelated rewrites, migration completeness
Maintainability	repository conventions, useful tests, understandable design
Reviewability	concise rationale, risks, assumptions and verification record
Recoverability	clean failure state, reversible change, no orphaned side effects

Hidden tests matter because an agent can overfit to visible checks. But do not make them arbitrary traps. They should represent real interfaces, edge cases and policies that a competent engineer could infer from the repository and task.

Score severe failures separately. A patch that solves the bug but introduces a critical authorisation bypass should not average into an acceptable result. Define blockers for secret exposure, destructive commands, unapproved network access, test tampering and evasion of required controls.

Observe the trajectory, not only the diff

The path to a patch reveals risks the final code can hide. Capture which files the agent read, what it searched, commands it ran, packages it installed, external resources it contacted, tests it altered and errors it encountered.

Useful trajectory questions include:

• Did it inspect repository guidance before editing?
• Did it reproduce the issue or guess at the cause?
• Did it broaden permissions to make the test pass?
• Did it delete or weaken a failing assertion?
• Did it retry a destructive command after uncertain completion?
• Did it notice generated files, migrations and documentation obligations?
• Did it stop when required context was unavailable?

The objective is not to enforce one reasoning style. It is to identify unsafe shortcuts and determine whether the evidence supports the result.

Put the agent inside a real security boundary

A coding agent may execute shell commands, read source, install dependencies, access networks and use repository credentials. Those capabilities turn evaluation into a security exercise as well as a quality exercise.

Run tasks in disposable workspaces with least-privilege credentials, restricted egress, resource limits and explicit secret handling. Seed tests with canary secrets and malicious repository content to see whether the agent follows untrusted instructions. Verify that it cannot write to protected branches or reach production services.

NIST’s Secure Software Development Framework organizes practices for protecting software, producing well-secured releases and responding to vulnerabilities. Apply those practices regardless of whether code was written by a person or agent. OWASP’s secure coding with AI guidance similarly recommends enforcing CI security checks and controlling the permissions available to agentic coding tools.

Evaluate the human review burden

An agent that produces many superficially plausible changes can increase reviewer load. Record the time and expertise needed to validate each patch, the number of review rounds, defects found only by humans and whether the change explanation directs attention to the real risks.

Give reviewers the same evidence they would receive in production. Do not show the gold patch. Ask them to decide whether to approve, request changes or reject. Measure disagreement between reviewers, because inconsistent acceptance may expose unclear engineering standards rather than agent variability alone.

Include deceptive-looking cases: a large polished refactor that misses the issue, a tiny correct patch, and a patch that passes tests by hard-coding the fixture. Review quality is part of the system result.

Test recovery from interruption and failure

Repository work rarely runs perfectly. Interrupt the agent during a dependency update. Make a package registry unavailable. Return a flaky test. Create an ambiguous merge conflict. Let a code-generation step partially complete.

The agent should leave a knowable workspace, distinguish its edits from pre-existing changes and explain what remains uncertain. It should not discard user work, conceal failing tests or claim completion after a timeout. Where side effects reach issue trackers or remote branches, require idempotent operations and reconciliation before retry.

NIST’s AI RMF says systems should be measured in conditions similar to deployment and monitored in production (AI RMF Core). For a coding agent, interruptions, dirty worktrees and imperfect tooling are deployment conditions—not optional stress tests.

Make the release decision repository-specific

Run a paired review before expanding permissions. Give the same task to the agent in a clean repository snapshot and in a realistic working copy containing unrelated user changes. The agent should isolate its work, adapt to the current state and never erase changes it did not create. Then repeat with a dependency outage and a deliberately misleading comment in the repository. These cases expose whether success depends on an unrealistically tidy environment.

Keep rejected patches in the evidence set. They reveal patterns that pass-rate summaries hide: unnecessary rewrites, shallow tests, ignored guidance or unsafe attempts to gain permission. A future model may solve more tasks while repeating a severe failure, so compare failure types as well as headline completion.

Report results by language, repository type, task family, consequence and permission set. A single pass rate hides whether the agent is strong on small test repairs and weak on migrations, or safe in libraries but unreliable in identity code.

Use the evidence to set a graduated operating envelope:

1. read and explain;
2. propose a patch in a disposable branch;
3. run approved tests and analysis;
4. open a review request with no merge authority; and
5. receive narrowly defined automation only for proven task classes.

Re-evaluate after changes to the model, harness, tools, permissions or repository. A useful software-engineering agent is not merely capable of producing code. It can operate within the engineering system, expose enough evidence for review and fail without putting the repository—or the people responsible for it—in an unknown state.