Portable evaluations are protection against platform lock-in

Vendor lock-in is usually discussed in terms of data export, proprietary APIs and contract exit. For AI agents, there is another asset that is easy to leave behind: the organisation’s definition of acceptable behaviour.

If the test cases, graders, traces and thresholds exist only inside one platform, moving elsewhere means more than replacing infrastructure. The buyer must reconstruct why the system was considered fit for use. A portable evaluation pack preserves that decision logic outside the supplier’s interface. It does not make migration free, but it turns part of the switching problem into an observable engineering exercise.

Own the standard of proof

The organisation operating the workflow should own the claims that matter: what the agent may do, what outcome counts as correct, which failures are critical, when a person must intervene, and what evidence supports continued operation. A platform can implement and help measure those requirements; it should not be the only place they are defined.

This is the evaluation equivalent of using interoperable formats. UK Open Standards principles connect open protocols and formats to easier migration, competition and sustainable cost. An evaluation pack extends that logic from technical exchange to assurance: the acceptance contract should remain intelligible when the underlying service changes.

Portability does not mean pretending platforms are identical. It means preserving the stable question while making platform-specific adapters explicit.

Define the portable core

A useful pack separates six layers:

Layer	Portable asset	Platform-specific adapter
Claims	Required outcomes, prohibitions and boundaries	Mapping to product controls
Cases	Inputs, initial state and expected constraints	Fixture creation and ingestion
Execution	Required action and environment conditions	Tool and agent invocation
Observation	Events and final state to capture	Trace and log extraction
Grading	Deterministic checks and review rubrics	Schema translation or grader API
Decision	Thresholds, uncertainty and response	Release workflow integration

Keep schemas documented and versioned. Use stable case identifiers that contain no confidential meaning. Preserve the original case source and authorised transformation separately from the runnable fixture. Store expected invariants—such as “no payment is created without valid approval”—rather than only a vendor-specific sequence of interface events.

The pack should be executable enough to expose ambiguity. A slide deck of requirements travels easily but cannot show whether two platforms satisfy them differently.

Grade external state before internal style

The most portable checks concern observable outcomes: the right record changed, a prohibited message was not sent, a handoff contained required facts, a transaction was not duplicated. Internal plans and proprietary reasoning traces vary across products and may not be available or appropriate to retain.

Use deterministic graders for structured state and policy invariants. Use rubrics with anchored examples for meaning, clarity or handoff quality. When a platform supplies an automated judge, keep a buyer-controlled sample graded by people and compare disagreements. A proprietary score without cases, version or calibration is not portable evidence.

NIST’s AI Risk Management Framework calls for documented test sets, metrics, methods and performance under conditions similar to deployment. That documentation is exactly what a portable pack needs. It also prevents a benchmark number from losing its meaning after the environment changes.

Test equivalence, not identical implementation

Suppose Platform A calls a refund API directly while Platform B submits a command to an approved queue. The traces differ, but both can be evaluated against the same outcome and authority claims. Create a canonical observation model containing the facts the decision needs: actor, authority, requested action, tool result, final business state, human intervention, elapsed time and cost.

Each platform adapter translates native evidence into that model and retains a link to the raw record where allowed. Validate adapters with known fixtures. If a field cannot be produced on one platform, do not fabricate equivalence; record a coverage gap and decide whether compensating evidence is acceptable.

This approach avoids reducing every product to the lowest common denominator. Teams may use a differentiated platform capability while keeping the decision rule independent. The portability question is not “can every implementation look the same?” It is “can we determine, using comparable evidence, whether each implementation met the same important requirement?”

Include cost and operating behaviour

Portability that measures only task success can recommend an alternative that is financially or operationally unusable. Run the same representative workload and collect distributions for latency, resource consumption, retries, human review, failures and quality-adjusted completion.

GAO’s review of cloud acquisition practices notes that interoperability, data portability and application compatibility help organisations maintain flexibility and avoid artificially increased costs. AI-agent evaluations should make the same concerns measurable. For example, an alternative may achieve equal outcome quality but require twice the manual review because it cannot expose a reliable confidence or policy signal.

Keep unit prices and pricing assumptions as versioned inputs rather than embedding them in the grader. This allows finance to rerun scenarios when contracts or usage change without rewriting the behavioural evaluation.

Rehearse the move while there is still leverage

A portability review is most useful before dependence becomes urgent. Select a small but representative slice of the pack and run it through an alternative implementation. Time the work required to:

1. export cases and permissible supporting data;
2. recreate tool contracts and policy boundaries;
3. execute and capture evidence;
4. translate observations;
5. grade and explain differences; and
6. produce a decision report.

The OECD’s study of data portability initiatives describes portability as a means of reducing switching costs and lock-in while improving interoperability and innovation. The European Commission’s Data Act explanation similarly highlights switching, open interfaces and machine-readable exports for data-processing services. Legal applicability depends on context and jurisdiction; the engineering lesson is broader: export must result in usable continuity, not merely possession of files.

Record the gaps the rehearsal finds. Perhaps the current platform owns the conversation state model, the grader depends on a proprietary trace, or the alternative lacks an action-control feature. Those are real switching costs. They can now be priced, designed around or consciously accepted.

Prevent the pack becoming another locked system

An internal evaluation platform can itself become a source of dependence. Keep the canonical case and observation formats simple, documented and exportable. Separate orchestration from definitions. Provide a command or documented procedure that produces a machine-readable result outside the user interface.

Version the following together:

• workflow and authority assumptions;
• case fixtures and provenance;
• environment and dependency versions;
• graders and calibration evidence;
• thresholds and decision owner;
• exceptions and unresolved coverage.

Protect sensitive fixtures with the same care as production data. Portability is not permission to scatter personal or confidential material across tools. Use synthetic or transformed cases where appropriate, and test secure transfer and deletion procedures.

Use portability to improve present decisions

The value appears before a switch. A platform-independent pack lets teams compare a new model, test a supplier release, investigate regression and challenge a claim using the same workflow standard. It makes procurement criteria concrete and gives product, engineering, risk and finance a shared decision object.

Not every agent needs a fully automated cross-platform harness. Proportionality matters. A low-consequence assistant may need a documented case set and exportable results. A high-dependence operational agent may justify maintained adapters and periodic alternative runs.

The strategic asset is not the ability to migrate overnight. It is the ability to explain what acceptable service means, reproduce the evidence on another implementation, and estimate the cost of change before the incumbent becomes the only organisation that knows how success is measured.