The way people imagine an artificial intelligence assistant failing is that it fails obviously. It refuses, or it produces something garbled, or it stops. You would notice.
That is not how it goes. The failure that costs you money is quiet, well formatted, and completely plausible.
Here is one that happened to me, with the actual numbers.
A card that said 3/11
I was building a small pipeline for a client, a print and textile designer running a studio business. She writes her expenses by hand on index cards as she goes, which is a good habit, and then the cards pile up because typing them in is nobody's favourite evening. Photograph a card, the expense lands in a spreadsheet. That was the job.
Before pointing anything at her real cards, I generated synthetic ones so the whole thing could be tested for free and repeatedly. Every number in this post comes from those test cards, not from her books.
One of them had a date written as 3/11. The reader came back with 2024-03-11, reported high confidence, and the row went into the expense sheet.
The real year was 2026.
Nothing raised an error. Nothing looked wrong. There was a correctly formatted date sitting in a financial record, two years off, filed with confidence. If that had been her actual bookkeeping, it would have sat there until something downstream did not add up, and then taken an afternoon to find.
Asking it to be more careful made it worse
The obvious fix is to tell it not to do that. Be careful with dates. Do not assume a year that is not written down.
So I tried that, and it produced a different failure. The model started refusing a perfectly legible 2/28/26, on the reasoning that it could not rule out the year 1926.
That is worth sitting with, because it is the shape of a lot of wasted afternoons. Tightening the instruction did not remove the problem. It traded a silent error for a useless one. The model was not being stupid in either direction. It was being asked to do a job that has no correct answer available to it, and filling the gap, first with a guess and then with a refusal.
The fix was structural, not verbal
The problem was never the wording. It was that one component was doing two different jobs: reading what was on the card, and deciding what it meant.
Those got separated. The model now returns a literal transcription of the characters it can see, and says nothing at all about what they signify. Then ordinary, boring, deterministic code does the interpreting: two digits means this century, and a card with no year at all gets the current one.
That last rule is a real assumption, and it could be wrong in January. So it is not hidden. It is written into a visible column that says the year was assumed, which means a human can scan for it instead of trusting it.
After the change, dates scored seven out of seven on the test set and the over-refusal disappeared.
The general version
Two rules came out of this that I now apply everywhere.
Never let the component that produces a value also decide whether the value is good enough. A reader that grades its own output is how a bad reading reaches a spreadsheet. Self-reported confidence is a feeling, not a measurement.
Separate transcription from interpretation. Ask the model for what it can literally observe. Do the reasoning in code you can read, test, and argue with. Anything that has one right answer given the input belongs in code, not in a prompt.
Neither of those is a clever trick. They are the kind of thing you only think to do if, before building anything, you wrote down how you would know the output was correct. That question is the entire method, and it is the one almost nobody answers unprompted.
So the three questions, which take about a minute and would have caught this on day one:
What must this produce. How will I know it is right. What must it never do.
Answer the second one properly and you will find the guess before it finds you.
Those three are the first half of the list I actually use before building anything. The full six, and the stop rule that goes with them, are on one page, free and with nothing to sign up for. The longer argument they came out of is here.