what breaks when agents meet a deal room

Six failure modes that show up in the first month of production.

We put agents into a live diligence workstream — an acquisition running four workstreams, with a data room growing by a few hundred documents a week. The mandate was narrow: draft first-pass summaries of contracts and flag change-of-control provisions for counsel. Nothing about the setup was exotic.

Six things broke in the first month, and none of them broke because the model was weak. They broke because a deal room is a moving, permissioned, half-structured environment and the agent was built as though it were a corpus. These are field notes rather than conclusions — but we have now seen most of them twice, and the fixes are cheap if you build them before the first incident rather than after it.

The first failure was version drift. A data room is a construction site, not a library. Documents are replaced, superseded, and re-uploaded under near-identical names, and an agent indexed on Monday is answering from Monday’s world on Thursday. Ours summarized a supply agreement that had been replaced two days earlier by an executed version with different termination language. The summary was accurate about a document that no longer mattered. The fix was to make freshness first-class: every document carries a version and a hash, the index is diffed against the room hourly, and any output built on a superseded version is flagged stale rather than left quietly standing.

The second was permission bleed. Workstreams had different access — the commercial team couldn’t see employment files — but retrieval had been pointed at the whole index. Nothing left the deal, yet an answer to the commercial team drew on a document they weren’t cleared to open. Entitlements existed at the folder level and were enforced nowhere in the pipeline. We moved the check to query time, so retrieval runs inside the asker’s access and a question that would need a document they can’t see comes back unanswered rather than answered.

Third: redlines. A marked-up agreement with tracked changes and three competing drafts of an indemnity clause is not prose, but it arrives as prose to a model handed extracted text. Ours produced a fluent summary of an indemnity position that existed in no version of the document — it had blended the seller’s proposed language with the buyer’s deletion. Confident, well organized, and wrong in a way only counsel would catch. Now documents with tracked changes or comparison markup are handled separately: the agent sees each version distinctly and is asked to describe the difference, not the document.

Fourth: retrieval that rewards length. Our initial scoring favored documents mentioning a term many times, so a two-hundred-page master services agreement outranked the four-page side letter that amended it. The side letter is almost always the thing that matters. We started weighting by document type and recency, and added a rule that any document referencing another pulls the referenced document into the result set. Diligence is a graph of amendments, not a pile of text.

Fifth, and the one we underestimated: reviewers stopped reviewing. Once outputs looked finished — clean headings, consistent formatting, citations in the right places — approval rates rose and review time fell at the same time. Polish reads as diligence. A junior clearing ten well-formatted summaries in an afternoon is not performing ten reviews.

We changed what review means. Outputs now surface what they couldn’t resolve — the clause the parser struggled with, the figure that appears twice with different values — and the reviewer signs against specific assertions rather than a whole document. A share of approved outputs is re-checked by a second reviewer, and the desk knows that happens.

Sixth: when the agent was wrong, nobody owned it. The first serious error produced half a day of polite confusion about whether it belonged to the vendor, the deal team, or us. Every agent now runs under a named desk owner, accountable for its outputs the way a supervisor is accountable for a junior’s work, with a written scope of what it may do unattended. That change did more for adoption than any improvement we made to the model, because it told the desk who was standing behind the answer.