Protocol conformance testing
Check implementation behaviour against a named protocol revision
A product needs evidence that required protocol behaviour is implemented. The decision is how to check implementation behaviour against a named protocol revision. The work begins with your actual workflow and looks for the evidence that separates intended behaviour from a consequential exception. It is particularly relevant to protocol implementers and quality leads. A useful outcome is an explicit account of what was examined, what remains uncertain, and which decision the evidence can support.
The interaction contracts relationship
- Normative requirements
Establish the basis for normative rule.
- Test obligation
Include negative and transition cases.
- Conformance evidence matrix
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around normative rule
Questions and the evidence they need
- Normative rule
- Inspect normative requirements for the relevant constraint or observation.
- Test obligation
- Inspect implementation settings for the relevant constraint or observation.
- Excluded behaviour
- Inspect test fixtures for the relevant constraint or observation.
The decision is how to check implementation behaviour against a named protocol revision. This is relevant to protocol implementers and quality leads when a product needs evidence that required protocol behaviour is implemented. Name the affected workflow, the consequence of a wrong decision, and the owner who can act on the result. The initial discussion should establish which evidence is already available and which uncertainty is material enough to investigate.
The specific failure hypothesis is that conformance is inferred from a working demonstration while error and cancellation requirements remain untested. It remains a hypothesis until examined in the deployment. Existing internal tests or operating records may already resolve it; identify the uncovered question before commissioning additional work.
Mechanism
Conformance evidence should map to selected requirements
Relationships within interaction contracts
- Normative rule
- Freeze revision and claimed feature scope.
- Test obligation
- Include negative and transition cases.
- Excluded behaviour
- State which requirements remain unexamined.
What this specialty examines
Conformance testing requires a named specification revision and an explicit selection of requirements. Separate mandatory behaviour, optional features and implementation extensions. Claims should indicate what was tested and what remained outside scope rather than treating a sample exchange as complete conformance.
The review should map selected requirements to positive, negative and state-transition cases for the frozen revision. Identify the normal route, relevant exceptions and the authoritative record of the outcome. Inspect the relevant business-system record, enforcement event or process evidence alongside any reported status. Keep incomplete cases visible, with an explanation of what their absence prevents the owner from concluding.
The tradeoff to make explicit
For this decision, precise conformance testing supports repeatability while business interoperability still needs separate evaluation. Compare the available design or operating choices against the same intended outcome. If one option removes a required control or changes the task boundary, describe that difference before treating its apparent simplicity, speed or lower effort as an improvement. The decision should expose the tradeoff instead of burying it in an aggregate score.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for normative rule
Illustrative working format
- Normative requirements
- Confirm the applicable revision and who resolves disputed interpretation.
- Implementation settings
- Check its relationship to the relevant workflow and expose missing context.
- Test fixtures
- Keep expected behaviour and the limits of the constructed example explicit.
- Claimed feature support
- Identify provenance and the specific decision this material can inform.
Start with normative requirements, implementation settings, test fixtures and claimed feature support. Identify the owner, revision and operating context of each source. Sanitised examples may support initial design, but differences from actual conditions should remain explicit. Where direct access is unsuitable, a customer-controlled evidence or execution route may be considered, with agreed handling and retention.
The scope also needs the integration owner, agreed expectations for a representative interaction case, and the authority to inspect the relevant boundary. Specify what evidence would distinguish the failure in which conformance is inferred from a working demonstration while error and cancellation requirements remain untested from acceptable behaviour. If stakeholders disagree about intended behaviour, resolve the interpretation before classifying the observed difference as a defect.
How the work proceeds
Carry normative rule through to an accountable finding
From the scoped material to a decision record
- Normative requirements
Agree the task meaning at the sending boundary.
- Test obligation
Apply the scoped check to excluded behaviour.
- Conformance evidence matrix
Record the evidence, open dependencies, and the owner’s next decision.
Fix the interaction contract and versions
Assemble normative requirements, implementation settings, test fixtures and claimed feature support with protocol implementers and quality leads. Identify the participating implementations, interface revisions, authority assumptions and intended business result. Prepare messages and task-state examples from both sides, including an interrupted exchange and a disputed or incomplete outcome. The preparation output is a paired contract fixture showing what each participant expects and which differences would prevent the workflow from completing correctly.
Run and compare both sides of the exchange
Map requirements to fixtures and observed outcomes, including error and state transitions. Preserve the exact configuration and any interpretation questions. A passing conformance case does not automatically prove compatibility with a particular business partner’s implementation.
Choose the contract or integration change
Summarise the comparison in a conformance evidence matrix with tested requirements and explicit exclusions. Classify each mismatch as interface validation, business meaning, identity, task state or operating behaviour. Decide whether to change a participant, add an explicit adapter rule, narrow supported combinations or defer the connection. Validate the affected pair again and preserve limitations that translation cannot resolve, especially around cancellation, authority and partial completion.
Indicative artefact
Make the conformance evidence matrix usable by its owner
Conformance evidence matrix
Illustrative working format
- Conformance evidence matrix
- Link observations and counterevidence to the case, configuration, and conclusion limit.
- Tested requirements
- Link supporting and contrary observations to inspectable evidence.
- Explicit exclusions
- Name the unresolved condition, next action, and accountable owner.
A useful decision artefact is a conformance evidence matrix with tested requirements and explicit exclusions. Supporting material may include selected cases, configuration or process records, evidence references, a disagreement log and prioritised actions. Retain the case selection rationale and material disagreements alongside the result; otherwise the result may be mistaken for complete coverage.
Illustrative scenario
Examine excluded behaviour through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Normative rule
An implementation parses valid messages but mishandles a required error transition.
- Test obligation
Exercise the specified error transition as well as valid parsing.
- Excluded behaviour
Report the unfulfilled requirement within the frozen feature scope.
Consider a fictional situation: a protocol endpoint accepts valid submissions but returns undocumented errors that consumers cannot interpret. The scenario is illustrative, with no client engagement, executed test or measured improvement implied.
For undocumented errors, compare observed responses with the selected requirement. The result may require an implementation correction or narrower feature claim. The consumer’s business recovery behaviour should then be evaluated separately from formal message conformance.
Scope and next action
Keep the conclusion within named implementations and revisions
Confirm fit before the next commitment
- Normative rule
- This review is limited to named implementations and revisions.
- Normative requirements
- The relevant owner must confirm access and the intended operating conditions.
- Conformance evidence matrix
- Untested peers require their own compatibility evidence.
Compatibility is assessed for named implementations, versions, features and workflows. It does not establish universal conformance, vendor endorsement or interoperability with untested peers. Any engagement depends on an agreed scope, qualified team, authorised access and confirmed availability. Bring the affected workflow, its current configuration or process, the triggering change and the evidence already available. The immediate scope question is prompted by the situation in which a product needs evidence that required protocol behaviour is implemented; confirm which missing observation would change the owner’s next action.
Ways to engage
Related engagement scopes
AI Agent Protocol Interoperability Lab
Find where connected agents disagree. Exercise message contracts, identity propagation and failure recovery across your selected integrations.
Continue exploring
Related insights and resources
How to design a joint AI evaluation lab
A joint lab should be a governed way to create decision evidence across organisations, with shared methods but explicit ownership of systems, data and conclusions.
Agent protocols are creating a new control surface
Protocols make tools and agents interoperable, but they also standardise discovery, delegation and context exchange. Those interfaces need explicit trust and policy controls.
A practical security review for an MCP server
Review an MCP server as a privileged protocol adapter: verify tool semantics, OAuth boundaries, token audience, downstream delegation, input handling, isolation, logging and consent.