capable is not reliable

A model that clears a task nine times in ten fails a desk that runs it fifty times a week. The fix starts at procurement — run the identical task many times, read the worst run, and route anything with a binary answer to code.

Take the task your vendor demoed — the covenant summary, the reconciliation narrative, the first-pass credit memo — and run it twice on the same desk, on the same morning, with the same prompt and the same documents. Do this before the contract is signed, not after. Often you will get two materially different answers: a figure pulled from a different table, a caveat present in one draft and missing from the other, a conclusion that hardened somewhere between ten o’clock and half past. Nothing changed except the run. Which means the demo was never showing you the product. It was showing you one draw from a distribution — and demos, like résumés, are assembled from the good draws.

What an institution deploys is not an agent’s best run. It is the distribution of its runs — the median, the tails, and the worst output the system will ever put in front of a client or a regulator. Capability is what the system can do when the draw goes well. Reliability is what it will do when the draw goes badly. Most evaluations measure the first. Everything that matters in production happens at the second.

The number a benchmark reports is the probability that one attempt succeeds. The number a desk lives with is the probability that every attempt succeeds, and the two come apart faster than intuition suggests. A model that clears a task nine times in ten looks strong on paper and fails a desk that runs it fifty times a week — at that rate the desk should expect around five bad runs every week, and the odds of a single clean week are well under one percent. Nobody on that desk experiences the ninety percent. They experience Tuesday’s failure, and then Thursday’s, and then the meeting about whether the tool can be trusted.

Finance has already internalized this distinction everywhere except here. Nobody procures a settlement system, a ledger, or a market data feed on its best day; a feed that is right nine times in ten is not ninety percent as valuable as a correct one, it is worthless, because every figure it emits now needs independent verification. The infrastructure of the industry is built out of floors — guaranteed worst cases — and the anomaly is that agents are currently being bought on ceilings.

The gap widens with every step you chain. If each step of a workflow succeeds ninety-five percent of the time — a rate that looks excellent in isolation — a twenty-step chain lands roughly one time in three, and the failures are not evenly distributed: an early misstep changes the context every later step reasons from, so errors propagate rather than average out. This degradation is invisible in precisely the two places institutions look before buying: short-task demos, which never chain enough steps for the compounding to show, and single-run evaluations, which by construction cannot see variance at all.

It is worth being clear about where the variance comes from, because the obvious fix does not work. Sampling is one source, but so are tool responses that differ between calls, a retrieval index that reordered itself overnight, and the ordering of documents in the context window — models are sensitive to the sequence in which they read, not just the content. Setting temperature to zero removes none of the last three and, in practice, not even all of the first; batching effects and floating-point arithmetic keep outputs from being bit-identical, and a system that is deterministic on identical inputs still diverges on the nearly identical inputs that production actually supplies. Zero temperature does not buy you determinism. It buys you a different lottery.

The procurement consequence is simple and, in our experience, almost never followed: run the identical task thirty or fifty times, and read the worst run, not the average. Then study how the errors cluster. If every bad run misreads the same clause, the failure is systematic — a guardrail or a parser fixes it. If the errors are scattered, uncorrelated, appearing in a different place each time, that is variance, and no prompt will remove it; only workflow design can absorb it. And distrust any evaluation — a vendor’s or your own — that reports one number per task, because one number per task means one run per task, and one run per task means the evaluation measured a draw, not a distribution.

The design consequence follows directly. Any step whose correctness is binary — a calculation, a reconciliation, a threshold check, anything transactional — should be routed to deterministic code, where the hundredth run is the first run by construction. The model earns its place where variance is survivable: drafting, summarizing, triage, the work where a bad run costs a review pass rather than a break. So the question to ask of any agent is not whether it can do the task; you already know it can, because the demo showed you. The question is what its hundredth run looks like — because the hundredth run, not the best one, is the one the institution actually lives with.