How Do You Know If AI Actually Did The Work?

Read the result, not the report, and ask what the tool checked rather than whether it is finished. Those two questions catch most of it. A summary is written by the thing being summarised, which is a conflict of interest we accept from software and would accept from nobody else. Open the actual file and count something. If the count and the summary disagree, believe the count.

Five Success Messages, None Of Them True

I ran five small jobs in a single session, each one making the same kind of change across a batch of files. Every one printed a tidy summary at the end. The summary was the only part any of them did reliably.

One reported twenty-four changes and had made thirteen. Another looked at twenty-five files, got eighteen of them wrong, and called them fine. A third destroyed work that was already correct, because it only saved at the very end of its run, and printed a success message on its way out.

Not one of them crashed. Zero errors, five times, which I had been quietly treating as good news. That is roughly the same reasoning as deciding your smoke alarm works because the house has never burned down.

It took until the fourth before I noticed they shared one bug. Every script was reporting what it set out to do rather than what it had done. Those are only the same sentence when nothing goes wrong, and something going wrong is the entire reason you are checking.

Why "Is It Done?" Is A Wasted Question

Asking whether something is finished has exactly one answer available to it. You will receive that answer whether or not it is true, and not because anything is being dishonest. The question invites a yes and offers nowhere for doubt to go.

On one job an AI told me the work was finished five separate times across two days. Four of those were honest. Each was true at the moment it was said, and then a deeper look turned up something genuinely new. That is decent work with a confidence problem sitting on top of it, and the confidence is what does the damage.

The fifth was a flat reversal. The same list of thirty-five open items, two opposite recommendations about an hour apart, delivered with identical certainty both times.

So ask a question with a different shape. What did you check, and what have you not checked yet? That one cannot be answered enthusiastically. You get a list instead of a reassurance, and a list is something you can make a decision with.

The Trap Underneath Both Of These

There is a version of this that is harder to spot, because the report is not wrong. It is incomplete.

One of my jobs sends the same large block of instructions on every single run. It looked like obvious waste, so I set out to redesign it. Before starting I printed one extra field out of the response, mostly so the plan would have a number in it. Almost all of it was already being discounted automatically, and had been the whole time.

The report had never lied. It had stopped one field short of useful, every run, for months. A wrong number eventually contradicts something and somebody notices. A missing number never contradicts anything at all.

The Question That Makes The Others Work

Run this over anything you currently rely on: what would this check show me if the thing were broken?

I once confirmed a website update had gone live by checking whether the page loaded. It loaded. The update had failed completely and the old page was loading exactly as it always had, which it would have done on any day of the week. My check could not tell success from failure, so it was never a check. It was a habit that felt like one.

Most of the reassurance people take from automated tools is this shape. The check runs, it comes back clean, and it would have come back clean regardless. If the answer to the question above is "it would look identical either way", you do not have a check. You have a ritual.

What To Actually Do

None of these need technical knowledge. Every one is a question you can ask out loud, and the awkward pause that follows is usually the answer.

Related reading: how to verify AI output before you act on it covers the layered version of this for data you cannot eyeball, and how to tell if AI research is any good is the same problem one step earlier, in the sources rather than the output.

---