# A rubric for a model that grades a spec Using a language model as a judge works when the judge has a rubric it can apply mechanically and a calibration set that proves it applies it the way you would. Without both, you have a second opinion of unknown quality, and you will trust it exactly when you should not. This file is the rubric, the prompt that wraps it, and the calibration procedure. It is written for grading specs and product requirements documents before an agent builds from them. Swap the dimensions for whatever you grade. --- ## The rubric Five dimensions. Each scored 0, 1, or 2. Anchors are written so two graders looking at the same document land on the same number. | Dimension | 0 | 1 | 2 | | --- | --- | --- | --- | | **Traceability** | Requirements state what to build with no source | Most requirements cite a source (interview, comment, sample, decision) but some are unsourced | Every requirement cites a source that a reader could open | | **Completeness** | A required section is missing | All sections present; at least one is a placeholder or a single line | All sections present and each carries enough to act on | | **Precision** | Fields, values, or behaviors described in prose only | Some fields typed and bounded, others described loosely | Every field has a type, a nullability, and a rule; every behavior has an example | | **Testability** | No acceptance criteria | Criteria exist but at least one cannot be checked mechanically | Every criterion is a check that passes or fails with no judgment | | **Decisiveness** | Open questions or TBD markers in the body | Open questions listed separately but still block implementation | No open questions, or the ones listed do not block the build | Total out of 10. Gate at 8, and require no dimension at 0. A document that scores 9 with a 0 in traceability is not a 9; it is a document you cannot audit. --- ## The judge prompt ``` You are grading a specification against the rubric below. Score each dimension 0, 1, or 2 using the anchor text exactly. For every score, quote the line or lines from the document that justify it. If you cannot quote a line, the score is 0 for that dimension. Do not rewrite the document. Do not suggest improvements unless asked. Your output is: 1. A table: dimension, score, quoted evidence. 2. Total out of 10. 3. Gate result: PASS if total >= 8 and no dimension is 0, otherwise FAIL. 4. If FAIL, the single dimension whose improvement would most raise the score, in one sentence. Rubric: Document: ``` The "quote the line" requirement is what makes this a grader instead of a vibe. A judge that has to cite evidence cannot award points for tone. --- ## Calibration, before you trust it 1. Pick ten documents you have already judged by hand: a few you would approve, a few you would send back, a few in between. 2. Score them yourself with the rubric, recording the number and the quoted evidence, before running the judge. 3. Run the judge on all ten. 4. Compare. Agreement within one point on every dimension is the bar. Where the judge disagrees by two, read its quoted evidence: either the anchor text is ambiguous (fix the rubric) or the judge misread (fix the prompt). 5. Keep the ten as a regression set. Rerun it whenever the rubric, the prompt, or the model changes. Ten is the minimum that catches a rubric problem. Twenty is comfortable. --- ## Failure modes to design against | Failure | What it looks like | Guard | | --- | --- | --- | | Grading its own work | The model that wrote the spec is asked to grade it | Different session, no shared context, or a different model | | Rubric drift | Scores creep up over months as everyone learns the rubric's words | The regression set; a document that scored 6 in January should score 6 in June | | Length bias | Longer documents score higher regardless of content | Anchors reward presence of specific artifacts, never volume | | Confident wrong quotes | Evidence cited that is not in the document | Verify a sample of quotes mechanically (string match) before accepting a run | | Gate gaming | Writers add a sources section that cites nothing real | Spot-check citations by opening them; make a fabricated source an automatic 0 | --- ## What the judge is for The judge is not the reviewer. It is the reason the reviewer's time is spent on documents that already clear the floor. A human still reads what passes. The gate removes the work of reading what should not have been submitted yet, and it removes the argument about whether it should have been, because the rubric is written down and the evidence is quoted.