Agent observability architecture
Decide what observability is necessary to explain a consequential agent action
When production behaviour cannot be reconstructed from existing logs, this area becomes relevant. For the SRE and AI platform lead, the concern is specific: the final answer is visible but the tool decision that caused the problem is missing. The engineering design starts from that consequence and follows the relevant evidence through the customer workflow. Design instrumentation from the questions an operator must answer. Capturing every token is not a substitute for recording the relevant decision, identity, configuration, tool effect, and handoff in a form reviewers can safely inspect.
The engineering decisions relationship
- Trace inventory
Establish the basis for correlation key.
- Authority attribution
Retain the mandate behind the operation.
- Observability architecture
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around correlation key
Questions and the evidence they need
- Correlation key
- Inspect trace inventory for the relevant constraint or observation.
- Authority attribution
- Inspect system identifiers for the relevant constraint or observation.
- Data minimisation
- Inspect evidence access and retention rules for the relevant constraint or observation.
For the SRE and AI platform lead, this becomes relevant when production behaviour cannot be reconstructed from existing logs. The immediate decision is what observability is necessary to explain a consequential agent action. The scope should be narrow enough that the engineering team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the final answer is visible but the tool decision that caused the problem is missing. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Mechanism
Observability should explain a consequential action
Relationships within engineering decisions
- Correlation key
- Connect events across tools and workflow components.
- Authority attribution
- Retain the mandate behind the operation.
- Data minimisation
- Capture useful evidence without unnecessary sensitive payloads.
Correlation across components
For this scope, correlation across components should be examined alongside trace inventory. Correlate events across request, agent, tools, and external effects. The proposed schema should preserve the identifiers needed to explain a material action. An attractive trace view is insufficient if reviewers cannot connect a tool call to the business record it actually changed.
Authority attribution
For this scope, authority attribution should be examined alongside system identifiers. Record the acting principal, authority decision, and relevant configuration. These details help distinguish an agent error from a permission or deployment change. The architecture should preserve evidence of the decision boundary without assuming that every raw prompt must be retained.
Sensitive-data minimisation
For this scope, sensitive-data minimisation should be examined alongside evidence access and retention rules. Minimise sensitive content while retaining useful investigation evidence. Define access and retention according to the customer’s requirements and qualified review. The observability design should avoid becoming an uncontrolled duplicate of private memory, documents, or credentials simply because more data might help debugging.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for correlation key
Illustrative working format
- Trace inventory
- Retain the observed conditions and distinguish gaps from contrary evidence.
- System identifiers
- Check its relationship to the relevant workflow and expose missing context.
- Evidence access and retention rules
- Confirm the applicable revision and who resolves disputed interpretation.
The starting material is trace inventory, system identifiers, and evidence access and retention rules. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a workflow owner, an implementation lead, an eventual maintainer, and qualified control reviewers where needed. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Carry correlation key through to an accountable finding
From the scoped material to a decision record
- Trace inventory
Make the component and operating boundary explicit.
- Authority attribution
Apply the scoped check to data minimisation.
- Observability architecture
Record the evidence, open dependencies, and the owner’s next decision.
Work backward from an investigation question such as why an update occurred. Identify the minimum events needed to connect request, principal, configuration, evidence, approval, tool request, and effect. The proposed telemetry design should include access and retention rules so observability does not become an uncontrolled copy of sensitive task data.
A practical first slice would focus on correlation across components and then follow its consequences into authority attribution. Keep sensitive-data minimisation visible as a separate review question rather than assuming it is satisfied by the first result. The engineering team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Indicative artefact
Make the observability architecture usable by its owner
Observability architecture
Illustrative working format
- Observability architecture
- Identify the assessed workflow, revision, and decision boundary.
- Event and trace schema
- Record accepted inputs, task meaning, and incomplete or failed outcome semantics.
- Investigation walkthrough
- Name the unresolved condition, next action, and accountable owner.
Indicative outputs include observability architecture, event and trace schema, and investigation walkthrough. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Illustrative scenario
Examine data minimisation through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Correlation key
An investigation asks why an agent reopened a case that a person had closed.
- Authority attribution
Correlate the human close event, agent mandate, and later reopen operation.
- Data minimisation
Retain sufficient attribution while minimizing unnecessary customer payloads.
Consider an illustrative situation in which an operations team investigates why an agent reopened a case after a human closed it. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the reopened-case example, an illustrative trace walkthrough would identify which component used the closed state and which later event changed it. Missing correlation would become an instrumentation requirement. The design should make this explanation possible without assuming reviewers can inspect every raw prompt or reconstruct timing from unrelated logs.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves sensitive-data minimisation unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Scope and next action
Keep the conclusion within the defined workflow and configuration
Confirm fit before the next commitment
- Correlation key
- This review is limited to the defined workflow and configuration.
- Trace inventory
- The relevant owner must confirm access and the intended operating conditions.
- Observability architecture
- Production acceptance needs separate scoped evidence.
The method described here is proposed engineering work. Scope, staffing, implementation access, acceptance, and commercial terms require a readiness review. A design or lab result does not establish production suitability, guaranteed reliability, or security certification.
Ways to engage
Related engagement scopes
AI Fractional Agentic AI Office
Create a regular place for agentic AI decisions. Connect initiative priorities, architecture questions, risk ownership and evidence reviews.
AI Fractional Agentic Transformation Office
Coordinate the operating changes behind agent adoption. Align workflow redesign, ownership, dependencies and adoption measures.
AI Harness Engineering Pod
Engineer a bounded AI execution harness with tool adapters, context handling, orchestration, recovery and observable traces.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
Governing agent decisions
Governance becomes useful when it changes a decision. Connect each agent workflow to an accountable owner, explicit operating conditions, proportionate evidence and a practical route for exceptions and withdrawal.
Do you need an agent centre of excellence?
A centre can concentrate scarce expertise and reusable controls, or become an approval queue detached from real work. Choose the mandate before the structure.