Why financial-services agents require more than model testing

A bank validates the model behind a service agent, checks response quality and signs off the model inventory. In production, the agent retrieves the wrong customer record, fails to recognise a dispute and sends a duplicate instruction after a payment timeout. None of those failures is adequately described by the model’s accuracy score.

A financial-services agent is a system of models, policies, data, tools, identities, interfaces, people and external providers. Model validation remains essential where applicable, but assurance has to follow the entire workflow and the outcomes it can create.

Start with the regulated service, not the model

Describe the service decision and consequence before cataloguing components. Is the agent answering a general question, initiating a payment, supporting underwriting, handling a claim, monitoring a trade or investigating financial crime? Who can be affected, which obligations apply and what recourse exists?

Then map the system:

• models and routing logic;
• customer and market data;
• retrieval and policy sources;
• internal and external tools;
• identities, permissions and credentials;
• human review and escalation;
• third-party services; and
• logging, monitoring and recovery.

The Financial Stability Board identifies model risk, data quality, cyber risk and third-party concentration among the vulnerabilities that AI may amplify in finance (FSB). Testing one model cannot establish that this connected system is safe or effective.

Connect model evidence to workflow evidence

Model tests answer questions about a component under defined conditions. Workflow tests ask whether the service produces an acceptable outcome when components interact.

Evidence layer	Example question
Model	Does the model perform consistently on representative language and tasks?
Retrieval	Did the agent select the current policy and correct customer context?
Decision	Did it apply exclusions, thresholds and escalation rules?
Action	Was the tool call authorised, correctly parameterized and confirmed?
Human	Could staff understand, challenge and override the recommendation?
Customer outcome	Was the request resolved fairly, accurately and with usable recourse?
Operation	Can the firm detect drift, incidents, dependency failure and harmful patterns?

The Bank of England PRA’s model-risk principles cover identification, governance, development, independent validation and mitigants, including relevant uses of AI in modelling (SS1/23). Firms should connect, not confuse, that discipline with broader agent-system assurance.

Test customer outcomes and rights recognition

Financial interactions contain language that can trigger complaints, disputes, hardship support, fraud handling or other controlled routes. Test indirect and distressed language, not only canonical phrases. Verify that the agent preserves the customer’s account and moves it into the correct process without inventing a resolution.

The CFPB has documented how financial chatbots can give inaccurate information, fail to recognise disputes and trap people without meaningful human help (chatbots in consumer finance). An agent with tools raises the stakes because it may also change an account or initiate a transaction.

Measure outcomes across relevant customer groups, products and channels. Include access to a person, explanation, correction and appeal. A high average answer score does not compensate for a small set of severe customer harms.

Evaluate authority and transaction state

List every action the agent can take and bind it to a workload identity. Define amount, account, product, timing and counterparty limits; required approvals; prohibited combinations; and expiry. Do not let the model decide its own permissions.

Test duplicate and uncertain execution. If a payment instruction times out, the agent must reconcile with the authoritative system before retrying. Distinguish proposed, approved, submitted, accepted, rejected, reversed and unknown states. Preserve an immutable link from customer request through policy check and approval to the external result.

Exercise attempts to bypass controls: split transactions, role changes, conflicting tool descriptions, untrusted instructions in retrieved documents and requests to expose another customer’s data.

Include third parties and operational resilience

Agents commonly depend on model providers, cloud platforms, data sources, identity systems and specialist APIs. Test what happens when each is slow, unavailable, changed or returns malformed data. Determine whether the firm can reduce authority, substitute a component, continue critical service and investigate an incident.

The BIS Financial Stability Institute says existing frameworks address many AI risks but highlights governance, skills, model risk, data governance and third-party providers as areas requiring attention (FSI Insights). Contracts and supplier reports are inputs to assurance, not substitutes for the firm’s own end-to-end tests.

Concentration matters too. Several apparently different agents may share the same model, data provider or identity service. Portfolio-level exercises should simulate that common dependency failing across services at once.

Make human oversight operational

Name the person or role accountable for each decision and exception. Define what evidence they see, how much time they have, what competence they need and whether they can reject or reverse the action. Sample actual reviews to detect rubber-stamping and hidden rework.

Do not treat a manual approval as a universal mitigation. High volume, poor interfaces and automation bias can make it ineffective. Compare the decision with and without the agent’s recommendation, and test whether reviewers notice deliberately inserted conflicts.

OECD analysis emphasizes governance, validation, monitoring and clear responsibility across AI use in finance (regulatory approaches). Those controls need evidence from the live operating arrangement.

Run continuous system assurance

Monitor component changes, policy freshness, data drift, action failures, customer complaints, overrides, downstream corrections and outcome disparities. Retain model and system versions so an incident can be reconstructed. Define thresholds for warning, authority reduction, route-to-human and shutdown.

Re-test after changes to models, prompts, tools, permissions, providers, product rules or customer journeys. A stable model score does not establish stable service behaviour when the rest of the system changes.

Rehearse one compound financial incident

Run an exercise in which the agent receives a customer’s ambiguous fraud report, retrieves a stale policy, experiences a tool timeout and then receives a plausible instruction from a compromised third-party document. The test should show whether the system recognizes the protected service route, rejects the untrusted instruction, reconciles transaction state and preserves evidence for investigation.

Include customer service, operations, security, compliance and the relevant model or technology owners. Each team should see the same timeline and be able to identify its decision. If the incident process depends on reconstructing events from separate vendor dashboards after the fact, that is an assurance finding. Exercise how authority is reduced while the service continues through a safe alternative.

Compare what monitoring detected with what the customer reported. A control that finds a duplicate transaction hours later may support recovery but does not prevent the initial harm. Record detection and containment time separately, and verify that customer correction does not depend on navigating the same failed agent.

Review whether the same signal could identify related cases across products without exposing more customer data than the investigation requires.

This is a technical assurance approach, not financial or legal advice and not a compliance opinion. The applicable requirements and supervisory expectations must be established for the particular institution, service and jurisdiction.

Financial-services agents require more than model testing because customers and markets experience the system’s actions, not its component score. The release decision should be based on evidence that the entire controlled service works—and can be stopped and recovered—within its intended scope.