# The evaluation plan

Six questions to answer before anything gets built with artificial
intelligence. This is the fill-in version of
https://michaelcarroll.studio/evaluation-plan, which is where the reasoning
behind each question lives.

Answer them in plain language, in a shared document, with the person who owns
the process in the room. Not in a ticket, and not alone. It takes about an
hour for one use case and it is the cheapest hour in the project.

Delete the guidance under each question once you have answered it, so what is
left is your plan rather than my template.

---

**Use case:**

**Date:**

**In the room:**

---

## 1. What must this produce?

*One or two sentences describing a thing, not an intention. "A first-draft
response to a supplier query, for a human to approve" is a specification.
"Improve supplier response times" is a hope, and nobody can evaluate a hope.*

>

## 2. How will we know it is right?

*The one that carries everything, and the one almost everyone skips because it
feels obvious until you write it down. Describe an input and the exact output
it should give, so somebody else could check it without asking you.*

*If the honest answer is that a person has to read every output and use
judgment, write that down too. It is still an answer, and it tells you the real
running cost before you commit rather than after.*

| Input | The exact output it should give |
| --- | --- |
|  |  |
|  |  |
|  |  |

**If you cannot answer this one, stop.** That is not a delay, it is the
finding. An unwritten standard is indistinguishable from no standard, and a
reviewer with no standard will correctly decline. Build nothing until this
question has an answer.

## 3. What must it never do?

*One-line rules, usually about money, personal data, or anything irreversible.
They go unwritten because they feel too obvious to say. They are not obvious at
all to a system that will otherwise fill the gap with something plausible.*

- Never
- Never
- Never

## 4. What counts as good enough to widen?

*A number, and the sample it is measured on, written down **before** you see any
results. Deciding what counts as success after you know the score is how
careful organizations talk themselves into shipping things they should not
have.*

- **Threshold:**
- **Measured on:** (how many cases, chosen how)
- **Set on:** (date, and confirm it was before any results existed)

*Include the number that actually predicts safety, which is not accuracy. It is
how often the system is wrong while looking right. A confident, well-formatted,
subtly wrong answer is the expensive failure, because nothing raises an error
and it is found a month later by somebody doing an audit.*

- **Wrong-while-confident rate, and the ceiling you will accept:**

## 5. Who signs this off, and what do they need to see?

*Name the person or the committee, and ask them now what evidence they would
need. This is the question that separates an organization where approval takes
a week from one where it takes two quarters, and asking it early costs nothing.*

*Most reviewers are not obstructive. They are being asked to accept a risk
nobody has described to them in terms they can weigh.*

- **Approver:**
- **Evidence they said they need:**
- **Asked on:**

## 6. What is explicitly out of scope?

*Write down what this is not, or it will grow while you are not looking. This is
the line that protects a fixed timeline, and the one most often left out.*

- Not
- Not
- Not

---

## Before you close this

Two checks that make the difference between a plan and a document nobody used.

1. **Did question 2 get a real answer, or a comfortable one?** If a stranger
   could not take your input-output table and check the system without asking
   you anything, it is not finished.
2. **Was the threshold in question 4 set before any results existed?** If not,
   it is not a threshold, it is a description of what you already got.

If you think one of these questions is wrong, or that there is a seventh, I
would genuinely rather hear it than not: https://michaelcarroll.studio/contact