How Do You Verify AI Output Before You Act On It?

You verify AI output by running checks that fail in different ways: deterministic arithmetic the documents already contain, a second model working blind with the results compared in code, and a human spot-check on a structured sample. Asking one AI to review another's work is not verification, because a reviewer that sees the first answer tends to agree with it. On a recent court deliverable, this layered pattern ran 602 field comparisons, surfaced 6 disagreements, and shipped 0 errors. The higher the stakes, the fewer layers you skip.

Key takeaways

• The model that did the work should never be the only thing that checks it. Same model, same blind spots.
• Deterministic checks beat model checks wherever they exist. Documents with printed totals audit themselves.
• A second AI must work blind. Showing it the first answer turns verification into anchored agreement.
• Compare outputs in code, not by asking a model whether they match. "602 fields compared, 6 disagreements" is evidence.
• Count everything. The closest this process came to failing was a batch that silently returned 7 of 9 records.

---

Why can't AI just check its own work?

Because the same reasoning that made the error reads straight past the error. When a model misclassifies overtime pay into the regular pay column, asking that model to review the spreadsheet finds nothing, since every number still looks right to the logic that put it there.

There are two failure modes in AI data extraction, and most people only guard against the first. Misreading is the obvious one: the model sees $2,773.42 and writes $2,773.24. Systematic misclassification is the quiet one: every number transcribed perfectly, and a whole category routed into the wrong column on every row. Spot-checking individual numbers never catches the second kind, because each number is "correct." It is just in the wrong bucket.

This is why independent verification appears in formal guidance too. The U.S. Government Accountability Office's accountability framework for AI systems centers on exactly this principle: the entity that built or ran the system should not be the only party assessing whether its output is right.

What does layered verification actually look like?

It looks like three checks that each catch what the others miss. The version below comes from a real project: 44 pay stubs extracted by AI into a court exhibit for a bankruptcy law firm, where the client asked the question every professional should ask. How do we know it's right?

Layer 1 is deterministic. Pay stubs, invoices, bank statements and tax forms all carry their own arithmetic: earnings lines sum to a printed gross, deductions sum to printed section totals, gross minus deductions equals net. Enforcing those identities during extraction means a misread digit becomes a loud failure instead of a quiet one. All 44 stubs had to balance before a single number entered the spreadsheet.

Layer 2 is where most verification goes wrong, so it gets its own section.

Layer 3 is cheap if you design for it. The delivered workbook included a verification tab with one row per stub: source file name, pay period, every extracted figure, and a math-check status. The client pulls three to five random stubs, compares them against their rows, and initials it. Five minutes, and a human has attested to the process. For legal work that layer is the one a court cares about.

What's the difference between agreement and verification?

Blindness. A second model that sees your filled spreadsheet anchors on your numbers and agrees with them. A second model that extracts from the raw documents on its own, with the two outputs compared field by field in code, is running a real test.

On the pay stub project, the second model never saw the first extraction. It chose its own columns and produced its own 14 fields per stub. Then a script compared the two sets: 602 comparisons, 596 exact matches, 6 disagreements.

Every disagreement was settled by opening the actual source document, not by voting. All 6 went the same way. The second model had missed retroactive adjustment lines printed at the bottom of the earnings section, one worth 62 cents and one worth $6.48. Without those lines the earnings did not sum to the printed gross, so Layer 1's arithmetic proved which extraction was right in minutes. This overlap is the point: the layers fail differently, so each one covers a blind spot the other can't see.

Research supports the intuition that models miss small details buried in long documents. A study by Nelson Liu and colleagues at Stanford, published in the Transactions of the Association for Computational Linguistics, found language model accuracy drops measurably when relevant information sits in the middle of a long input rather than at the edges. Fine print at the bottom of an earnings table is exactly the kind of detail that goes missing.

When is this level of checking worth it?

When someone else wears the consequences of a wrong number. Court exhibits, financial reporting, tax filings, medical billing, compliance work, anything a client signs or a regulator reads. The NIST AI Risk Management Framework makes the same stakes-based argument: the rigor of measurement and oversight should scale with the harm a wrong output can cause.

For low-stakes, reversible work, skip layers freely. A blog draft doesn't need a blind second extraction. My own rule from the court project: skip whichever layers the stakes allow, and for anything going in front of a judge, skip none.

The pattern also fits a bigger habit worth building: treating confident AI output as a claim rather than a fact. I keep a four-pass checklist for pressure-testing AI research for the same reason, and the memory files described in how to make AI remember your business between sessions exist partly so those verification rules survive from one session to the next.

---

Frequently asked questions

Can't I just ask a second AI to review the first one's work?

You can, and it will usually agree, which is the problem. A reviewer that sees the first answer anchors on it. The fix costs nothing extra: give the second model the raw source material instead of the finished work, let it produce its own answer, and compare the two outputs in code. Agreement between two blind, independent extractions means something. Agreement with a visible answer mostly measures politeness.

How much time does layered verification add?

Less than it sounds like. The court exhibit verification took about an hour and a half to design and run, and most of that was waiting on API calls. The arithmetic layer runs during extraction, the diff is a script, and the human spot-check is around five minutes with a prepared verification sheet. Against a deliverable someone must defend in court, that is a small cost.

What happens when the two models disagree?

Open the source document and settle it there. Never resolve a disagreement by majority vote or by trusting the more confident model. On my project, all 6 disagreements went to the same side once the actual pay stubs were opened, and a vote would have gotten at least one wrong if a third model had happened to share the second one's blind spot.

Does this only apply to data entry?

No. The principle carries to any AI output you act on: research reports, competitor analysis, financial summaries, legal drafts. Find the deterministic checks that exist in your material, get an independent second read where the stakes justify it, and keep a human attestation step for anything with consequences. The layers change shape by task. The idea that checks must fail differently does not.