In Evaluation Is the New Code Review I said that designing the tests is the hard skill in spec-driven development. One of the quality gates I run is a large language model (LLM) acting as a judge, scoring every spec before an agent builds from it. People hear that and picture AI grading AI, which sounds like a loop with no floor. The reason it works has nothing to do with the model. It is the rubric, and the calibration that proves the rubric is being applied the way I would apply it.
Anchors, Not Adjectives
A rubric that says "clarity: 1 to 5" is an invitation to grade on tone. Mine has five dimensions, each scored zero, one, or two, and each score has anchor text specific enough that two graders reading the same document land on the same number.
| Dimension | What a two looks like |
|---|---|
| Traceability | Every requirement cites a source a reader could open: an interview, a comment, sample data, a decision |
| Completeness | Every required section is present and carries enough to act on; no placeholders |
| Precision | Every field has a type, a nullability, and a rule; every behavior has an example |
| Testability | Every acceptance criterion is a check that passes or fails with no judgment |
| Decisiveness | No open questions, or none that block the build |
Ten points total. The gate is eight, and no dimension may be zero. That second condition matters more than the first: a document that scores nine with a zero in traceability is not a nine, it is a document you cannot audit.
Quote the Line
The single most important instruction in the judge prompt is this: for every score, quote the line or lines from the document that justify it, and if you cannot quote a line, the score is zero. That one rule turns a vibe into a grader. A judge that has to cite evidence cannot award points for confidence, length, or a well-written introduction. It also produces a report I can check in thirty seconds, because the evidence is right there and either it is in the document or it is not.
Calibrate Before You Trust
Nobody should run a judge they have not tested against themselves. The procedure is short. Pick ten documents you have already judged by hand, a few you would approve, a few you would send back, a few in between. Score them with the rubric yourself first, evidence and all. Run the judge. Compare. Agreement within one point on every dimension is the bar. Where the judge is off by two, read its quoted evidence: either the anchor text is ambiguous and the rubric needs fixing, or the judge misread and the prompt does. Then keep the ten as a regression set and rerun it every time the rubric, the prompt, or the model changes.
Five Ways It Goes Wrong
| Failure | Guard |
|---|---|
| The model grades its own work | A different session with no shared context, or a different model |
| Scores creep up as writers learn the rubric's words | The regression set; a six in January is a six in June |
| Longer documents score higher | Anchors reward the presence of specific artifacts, never volume |
| Quoted evidence that is not in the document | String-match a sample of quotes before accepting a run |
| A sources section that cites nothing real | Spot-check by opening them; a fabricated source is an automatic zero |
What the Judge Is For
The judge is not the reviewer. A human still reads everything that passes, and the human checkpoint on a merge is not going anywhere. What the judge removes is the work of reading what should not have been submitted yet, and the argument about whether it should have been, because the rubric is written down and the evidence is quoted. Reviewer time gets spent on documents that already clear the floor.
The rubric, the prompt, the calibration procedure, and the failure table are in judge-rubric.md. Swap the five dimensions for whatever you grade and keep everything else.
This is part of a track on agent tooling and spec-driven development that goes deeper than a field report, with something you can download and run in each one. Free for now while the track grows; the deepest installments become a paid tier later.