evaluating financial reasoning

Building a harness that scores judgment rather than fluency.

Most published benchmarks ask a model to produce an answer and then check whether the answer matches a key. That works for arithmetic and for questions with one defensible response. It works badly for the work that actually fills a credit committee’s afternoon — whether a borrower’s EBITDA add-backs are aggressive, whether a covenant headroom calculation holds under a plausible downside, whether the disclosure in a Q3 filing changes how you read the guidance. In those cases the answer is a recommendation, and the recommendation is worth exactly as much as the reasoning that produced it. A model that gets the conclusion right by fluent guessing is not useful; it is a liability that has not yet been discovered.

So when we install agents inside a bank or an asset manager, the first thing we build is not the agent. It is the harness — the thing that decides whether the agent’s reasoning would survive contact with the people who currently do the work. That harness is built from the institution’s own record: the memos it wrote, the decisions it made, and, crucially, what happened afterward. It is unglamorous, it takes weeks, and it is the single artifact that determines whether anything downstream gets into production.

A generic finance benchmark tells you how a model performs on a public corpus that has almost certainly been part of its training. It tells you nothing about how it handles the way your credit team words a waiver request, or the three exceptions your investment committee has quietly carried for a decade. We build eval sets from the firm’s own decision history instead — a few hundred cases pulled from past credit memos, investment committee packets, and covenant compliance reviews.

The valuable part of that history is not the decision. It is the pairing of the decision with what happened next. A loan approved in one year that went to workout two years later, an issuer downgraded six months after the analyst called it stable, a covenant reset three times before anyone asked why — these are the cases where judgment was tested and either held or did not. We label them with the desk, not around it. The analyst who wrote the memo is usually the only person who can say which of the twelve factors listed actually moved the decision.

The set does not need to be large. It needs to be adversarial in the right places — heavy on the marginal cases, where the file was genuinely ambiguous and reasonable people disagreed. Easy cases tell you nothing, because everything passes them.

Once you have the cases, the temptation is to score the output — approve or decline, in-range or out. We score the chain instead. For a covenant check, that means asking whether the agent found the right definition in the right amendment, applied it to the right period, used the right adjusted figure, and carried the calculation without silently substituting a more convenient number. Each of those is a separate assertion that can be checked against the source. An agent can reach the right conclusion through four wrong steps that happen to cancel. In production, they stop canceling.

This changes what a failure looks like. A model that declines a facility for the wrong reason has failed, even though the recommendation matches the committee’s. Scoring at the step level makes failures diagnosable — you can see whether the problem is retrieval, arithmetic, or the harder thing, which is knowing which of two conflicting documents governs.

The other axis is support. The failure mode that matters most in a regulated setting is not being wrong; it is being wrong in a register that reads as certain. A sentence asserting that a borrower is in compliance as of the September test date, written when the September financials were never delivered, will move through an institution unchallenged because it sounds exactly like the sentences that are true. Every material claim has to trace to a document in the context graph, and untraceable claims are penalized in proportion to how confidently they are stated. A refusal where the file does not support an answer scores well — which is not how most benchmarks treat it, and is the whole point.

A new frontier model arrives every few months. Without a harness, each arrival restarts the trust conversation from zero — the risk committee has to be persuaded again, model validation opens a new file, and the desk spends a quarter re-establishing what it already knew. That is why firms end up frozen two generations behind on a model they never quite trusted in the first place.

With a harness, the upgrade becomes an operation rather than a negotiation. You run the new model against the same few hundred cases, compare step-level scores against the incumbent, and look specifically at what regressed. Sometimes the newer model is better overall and worse on the one thing the desk cares about most, which is exactly the finding you need before anything is switched.

The harness is also the thing that survives us. The agents will be replaced and the models certainly will be, but the institution’s record of what good reasoning looked like on its own hardest cases is durable — and no vendor can supply it, because it is made entirely of the firm’s own decisions.