LLM audit trail requirements for financial services

Financial services firms need a call level audit trail of LLM usage: a record of every request showing which team sent it, which data reached which model, what it cost, which key authorized it, and what happened when a provider failed. Examiners now ask for the usage log, not the policy, and the only architecture that reliably produces that log is a single gateway in front of every model provider.

This shift catches many firms off guard. AI governance programs often start with policy documents, committees, and training. Those matter, but none of them generate the record an examiner reviews.

The examiner asks for the log, not the policy

Bank examiners are trained to test whether a control operates, not whether it is written down. An AI use policy that prohibits sending customer data to public models is a claim. The usage log is the evidence. When a firm cannot produce a record of which prompts went to which providers, the conversation shifts from a review of AI adoption to a finding about missing controls. Teams that treat LLM logging as an engineering preference rather than a supervisory record usually discover the difference during an exam.

The four areas examiners probe

1. Cost attribution by team. Finance and risk both need to know which team spent what across which models. Shared keys and pooled invoices make attribution impossible and turn budget reviews into guesswork.
2. Call level audit trail. For each request: timestamp, calling team or application, model and provider, the classification of data in the payload, token counts, cost, and outcome. This is the record that answers which data reached which model.
3. Key custody and revocation. Who holds provider keys, where they are stored, and how quickly access can be revoked when an employee leaves or a key is exposed. Keys pasted into application code fail this test.
4. Provider failover. What happens when a model provider degrades or goes down. Examiners treat this as an operational resilience question, and they expect a documented, tested answer.

Why direct provider integrations cannot produce the record

When each application integrates directly with each provider, the audit trail fragments. One provider's dashboard knows only the traffic that reached that provider. No provider can log what it never saw, attribute usage to your internal teams, apply your data classification, or record the failover decision that moved a request from one model to another. Reconstructing a unified record from separate provider dashboards after the fact is not an audit trail. It is an explanation of why you do not have one.

The structural fix is one gateway in front of every provider

Route all LLM traffic through a single gateway that your firm controls. Every request passes one checkpoint, so logging, key custody, budgets, and failover are enforced in one place instead of being reimplemented in every application. Self hosted gateways such as LiteLLM run inside your own VPC, which keeps prompts and logs inside your network boundary while providing per team keys, per team budgets, call level logs, and failover across 100+ providers. The gateway becomes the system of record the examiner asks for, and it exists from the first request onward rather than being assembled under deadline.

Frequently asked questions