Agent evaluation centre of excellence
Decide what a centre of excellence should own and what teams should retain
When evaluation knowledge is scattered across product teams, this area becomes relevant. For the AI evaluation or platform leader, the concern is specific: central standards become detached from the tasks and consequences of individual workflows. The operating model design starts from that consequence and follows the relevant evidence through the customer workflow. A centre of excellence should help teams ask better questions and maintain reusable methods. It should not imply that one central score can settle release decisions across different business contexts.
Connect the operating question to inspectable evidence
- AI evaluation or platform leader
- Owns the bounded decision and the consequence of acting.
- Evaluation methods
- Defines the starting context for the proposed work.
- Method reuse
- Exposes the first material uncertainty to examine.
- Adoption and calibration plan
- Carries unresolved conditions into an accountable next action.
Decision at stake
Decide what a centre of excellence should own and what teams should retain
Make the decision explicit
- AI evaluation or platform leader
- Evaluation knowledge is scattered across product teams.
- Decision
- What a centre of excellence should own and what teams should retain.
- Method reuse
- Use evaluation methods to establish the relevant starting condition.
For the AI evaluation or platform leader, this becomes relevant when evaluation knowledge is scattered across product teams. The immediate decision is what a centre of excellence should own and what teams should retain. The scope should be narrow enough that the customer leadership team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that central standards become detached from the tasks and consequences of individual workflows. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine method reuse alongside reviewer calibration
Keep the review distinctions visible
- Method reuse
- Reuse cases only when their assumptions fit the local deployment.
- Reviewer calibration
- Compare reviewer judgements on the same ambiguous evidence.
- Local decision accountability
- Keep product release authority with the local accountable owner.
Method reuse
For this scope, method reuse should be examined alongside evaluation methods. Maintain shared methods, fixtures, case conventions, and finding rubrics where reuse is credible. The centre should identify who updates these assets and how changes are reviewed. Reuse should support teams without erasing the task-specific requirements that determine their release decisions.
Reviewer calibration
For this scope, reviewer calibration should be examined alongside case and rubric libraries. Calibrate reviewers on representative synthetic findings and preserve disagreements. Different interpretations can reveal an unclear rubric or missing domain context. The proposed process should improve consistency through evidence and discussion rather than assume one automated score removes judgement from evaluation.
Local decision accountability
For this scope, local decision accountability should be examined alongside product team responsibilities. Keep product owners accountable for local acceptance criteria and release consequences. A central function can challenge and support the evidence chain, but cannot assume every workflow has the same definition of success. Adoption should be demonstrated through maintained practice rather than template downloads alone.
Prerequisites
Start with evaluation methods
Give each input a specific job
- Evaluation methods
- Defines the operating boundary.
- Case and rubric libraries
- Examine reviewer calibration.
- Product team responsibilities
- Shows conditions affecting local decision accountability.
- Executive sponsor
- Confirm access and a reviewer who can resolve gaps.
The starting material is evaluation methods, case and rubric libraries, and product team responsibilities. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include an executive sponsor, accountable delivery owners, and the people who can accept or implement the resulting decisions. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Make the mandate usable in operating decisions
Follow the evidence through the work
- Method reuse
Use evaluation methods to define the decision criteria.
- Reviewer calibration
Inspect case and rubric libraries and retain observations that challenge the expected result.
- Local decision accountability
Use product team responsibilities to identify unresolved conditions and a responsible owner.
- Adoption and calibration plan
Link the recommendation to its evidence, remaining limits and next validation condition.
Separate shared method ownership from product-specific acceptance. A central function can maintain case conventions, finding rubrics, reusable fixtures, and reviewer calibration, while product teams define task outcomes and release consequences. The proposed model should provide a way for local failures to improve shared assets without exposing sensitive evidence.
A practical first slice would focus on method reuse and then follow its consequences into reviewer calibration. Keep local decision accountability visible as a separate review question rather than assuming it is satisfied by the first result. The customer leadership team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Indicative artefacts
Use adoption and calibration plan to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Evaluation operating model
- Boundary — evaluation methods.
- Shared asset governance
- Findings about reviewer calibration.
- Adoption and calibration plan
- Action on local decision accountability.
Indicative outputs include evaluation operating model, shared asset governance, and adoption and calibration plan. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Illustrative scenario
Shared conventions, local criteria: calibration learning
Examine the constructed decision boundary
Illustrative scenario
- Shared conventions, local criteria
Agent teams want common evidence methods while retaining release accountability.
- Same synthetic finding
Ask reviewers to interpret identical evidence and expose disagreement.
- Calibration learning
If judgements diverge, improve the method before treating template downloads as adoption.
Consider an illustrative situation in which several agent teams want common evidence conventions while maintaining their own release criteria. This is a constructed example, not a client engagement or an observed Softlander outcome.
For teams with different workflows, an illustrative calibration session would compare how reviewers interpret the same synthetic finding. Disagreement would become a method improvement question. The centre should be judged by adoption and decision usefulness, with evidence that teams can apply and update the method rather than merely download a template.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves local decision accountability unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Scope and authority
A proposed mandate needs confirmed authority and capacity
Keep analytical work separate from decision authority
- Operating mandate and handover design
- Can include evaluation operating model.
- Executive sponsor
- Customer executives retain investment, release and risk decisions.
- Staffing, price and retained commitments
- Cannot be inferred from shared asset governance.
These are proposed working arrangements, not claims of current staffing or an active retained relationship. Executives retain investment, release, and risk decisions. Capacity, role authority, conflicts, and commercial terms need explicit agreement before a commitment.
Ways to engage
Related engagement scopes
AI Fractional Agentic AI Office
Create a regular place for agentic AI decisions. Connect initiative priorities, architecture questions, risk ownership and evidence reviews.
AI Fractional Agentic Transformation Office
Coordinate the operating changes behind agent adoption. Align workflow redesign, ownership, dependencies and adoption measures.
AI Harness Engineering Pod
Engineer a bounded AI execution harness with tool adapters, context handling, orchestration, recovery and observable traces.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
Governing agent decisions
Governance becomes useful when it changes a decision. Connect each agent workflow to an accountable owner, explicit operating conditions, proportionate evidence and a practical route for exceptions and withdrawal.
Do you need an agent centre of excellence?
A centre can concentrate scarce expertise and reusable controls, or become an approval queue detached from real work. Choose the mandate before the structure.