The Hidden Risk In Automating Financial Models With AI
Aritra Pal, CFA, builds machine learning and risk models for regulated banks, with 17+ years in treasury and capital markets tech.
gettyEvery bank I’ve worked at wants to move faster: credit decisions due the same afternoon, forecasts ready before the quarter they cover. I’ve spent more than a decade building machine learning models inside regulated financial institutions, and the failure I’ve come to fear isn’t a wrong number. It’s a model still running smoothly after everyone who understood it has left.
I’ve watched it happen. A model my team built ran in production for months without incident. Then a routine validation review raised a question about a design decision made in the project’s first week, a choice about how certain inputs were structured to work around a gap in the source data. Nobody could fully explain it. The decision made sense at the time, but it was never written down, and the person who made it had since moved to another team. Without that reasoning, we couldn’t satisfy the reviewer, and the model came out of production.
I still think it was a reasonable model. That opinion didn’t count for much. What stayed with me wasn’t really the loss of that one project. It was how little time it took for that knowledge to disappear. According to Gartner, “through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.” The reasoning behind a model tends to live in people’s heads rather than in documentation; people move on to other teams, and unless someone deliberately holds onto that reasoning, it leaves with them.
The first question I usually get from leadership on a new AI initiative is which algorithm we’re using. Almost nobody asks where the data comes from, and that’s usually where most of the real work sits. Pulling consistent, reliable data out of six or seven internal systems that were never built to talk to each other takes far more time than most people expect.
One system posts its updates overnight. Another runs a full day behind, and nobody has documented that it does. Every system carries a quirk like that, and someone has to notice it and correct it by hand, usually the same person each time, working from memory rather than a written process.
Automation raises the stakes on all of this. A bad input turns into a bad output faster, with fewer people positioned in the middle to catch it before it reaches a decision. A leader who signs off on an AI initiative without asking pointed questions about the data feeding it is taking on more exposure than the steering committee slide ever shows.
On one project, we tested a heavier, more sophisticated model against a simpler one. The heavier version won, but only by a fraction of a percentage point, and that small edge came at a real cost: It took most of an afternoon every time we had to walk a risk committee through how it actually arrived at a number. We shipped the simpler model instead and kept the more complex one as a challenger to monitor over time.
In a regulated environment, the people accountable for an institution have to be able to sit across from a validator and defend exactly what’s running. A model nobody can fully explain doesn’t clear that bar, no matter how well it scores on a benchmark. I’ve come to treat that as a design constraint that shapes a model’s architecture from the beginning rather than something discovered the hard way during a review.
Most of what changed for me is unglamorous, and the most important piece is just writing decisions down as they’re made, including the ones that feel too small to bother with. Nearly every governance problem I’ve run into traces back to a choice that made complete sense in the moment and none at all once the person who made it had moved on.
That first-week decision about how the inputs were structured was exactly this: a sensible call that lived only in one person’s head, and its absence from any document is the whole reason a working model came out of production. A two-line note at the time would have saved weeks of untangling later.
Explainability belongs in the same place, written into the design document from day one, next to the accuracy targets rather than bolted on before an audit. The test I use is whether a reviewer five years out, someone who never met the team, could follow the model’s reasoning from what’s written down alone. If they couldn’t, the work isn’t done yet.
Monitoring deserves the same forethought. A forecast that looks solid this month can begin drifting the next, and by the time the drift is obvious in the numbers, it has usually been underway for a while. The teams that catch it early are the ones who decided ahead of time what they’d watch and when they’d act.
And then there’s the data work, which needs a real budget and needs it early. It’s the first line finance tries to trim because it doesn’t look like much on a slide, and it’s the thing you’ll miss most when something breaks in a way nobody notices until later.
One last habit is more personal than institutional. Every so often I take one of my own past decisions and try to defend it cold, no warning, the way a validator eventually will. I fail at that more often than I’d like to admit, and I keep doing it anyway.
You can automate a model in a matter of weeks. Keeping it explainable isn’t a phase that ends at launch; it runs for as long as the model itself does, and that work rarely gets a ticket of its own. Someone, eventually, has to stand up and defend what’s running, long after the people who built it have moved on.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
