Every conversation I have about AI in a finance function eventually arrives at the same place. The model works. The accuracy is demonstrably better than the manual process. And the CFO still won't sign it off.
This gets read as conservatism, or as a change management problem to be solved with a workshop. It is neither. The CFO is asking a question the system genuinely cannot answer: when someone asks in eighteen months how this number was produced, what do we show them?
Accuracy is not the barrier. Explicability after the fact is.
What "show your working" actually means
Technical explainability — feature importances, attention weights, SHAP values — answers a question auditors are not asking. Nobody wants to know which input features the model weighted most heavily. They want a chain that runs from a number in a set of accounts back to source documents, through every transformation, with the identity of whoever or whatever made each judgement.
That is a records problem, not a machine learning problem. Which is good news: it is a solved category of problem, provided you design for it up front. Retrofitting it is close to impossible, because the information you need was never captured.
Four things have to be reconstructible for any given output:
- Inputs. Which exact records, at which versions, as they existed at the time — not as they exist now.
- Logic. Which model version, which prompt, which configuration, which thresholds.
- Judgement. What the system concluded, with what confidence, and which alternatives it considered and rejected.
- Human involvement. Who reviewed it, when, what they saw at the time, and what they changed.
The four layers
1. Lineage
Every output traces to specific input records, identified by version. This is where most systems fail immediately, because source data is mutable. If a supplier record was corrected last Tuesday, and your reconciliation ran the Tuesday before, then re-running it today produces a different answer and you cannot demonstrate the original was right.
Append-only source records, or point-in-time snapshots. Whichever you choose, the requirement is that you can reconstruct the state of the world as the system saw it, not as it looks now.
2. Decision logging
Log the reasoning, not just the outcome. For each decision: inputs consulted, confidence score, the conclusion, and the alternatives that were rejected. That last element gets skipped constantly and it is the one auditors find most persuasive, because it demonstrates the system considered the case rather than pattern-matched it.
Log at a stable granularity. If you log per-batch, you cannot answer a question about one transaction — and questions always arrive about one transaction.
3. Human checkpoints, placed on purpose
Rubber-stamping is worse than no review at all: it creates a signature that implies scrutiny that never happened. Route only what genuinely needs judgement — the uncertain band — and give the reviewer enough context to exercise it.
Critically, capture what the reviewer saw, not just what they decided. An approval is only evidence if you can show what was in front of the person when they gave it.
4. Reproducibility
Given an output and its logged inputs, you should be able to re-run and get the same answer. With non-deterministic models this needs deliberate work — pinned versions, recorded parameters, stored intermediate outputs where exact reproduction isn't achievable.
The honest position when full determinism isn't possible: log enough that you can demonstrate the decision was reasonable given what the system saw, even if the exact output cannot be regenerated. Auditors accept this if you explain it in advance. They react badly to discovering it during an audit.
Retrofitting an audit trail is not hard. It is impossible — the evidence you need was never captured, and no amount of engineering afterwards can create it.
What this costs
I want to be straight about the trade-offs, because pretending they don't exist is why these conversations go badly.
Storage grows substantially — you are keeping every input version and every decision record, and the log will often exceed the operational data. Latency increases, though rarely enough to matter for batch work. Development slows, perhaps by a quarter to a third, because instrumentation has to be built alongside the logic rather than after it.
Against that: it is the difference between a system that goes into production and one that doesn't. A pilot that cannot be audited stays a pilot indefinitely, which is a total loss of the investment rather than a 30% overhead on it.
How to have the conversation
If you are trying to get a finance function over the line, three things have worked for me.
Involve audit before you build, not at the end. Ask what evidence they would need. They will tell you, in detail, and it is usually less than engineers assume. What they want is completeness and consistency, not sophistication.
Demonstrate with a real trace. Take one transaction and walk it end to end — source documents, transformations, decision, confidence, reviewer, final figure. One concrete example ends the conversation faster than any amount of architecture discussion.
Start where the stakes are low. Run the system in parallel with the existing process for a period. It generates an audit trail nobody has to rely on yet, and by the time you want to rely on it, you have months of evidence that it holds up.
The organisations getting AI into regulated processes aren't the ones with the best models. They are the ones that treated evidence as a design requirement rather than a compliance afterthought.