Who validates the validators?

The judge is a model.

At any real scale, you're running an LLM-as-judge: a model that reads the product's output and returns a verdict. The judge has a rubric — the things to check for — and a verdict — pass, fail. You trust the verdicts because you wrote the rubric, and the rubric encodes your standards.

Except the judge is a model. It misreads rubrics. It hallucinates rationales. It's slightly biased toward whichever output happens to be listed first. It does the things LLMs do when nobody's watching, and it does them quietly, because the output is just a true/false and a one-line reason.

Shreya Shankar wrote a paper about this: Who Validates the Validators? The thesis is simple and corrosive. If you don't validate the judge against humans, you don't know what the judge is measuring. You assume it's measuring the rubric. It's probably measuring something correlated with the rubric, and also with the model's priors, and also with how the prompt was formatted. The correlations drift from your rubric silently, and the drift is invisible unless you look.

The validation move.

Hold back a set of outputs humans have already graded — labeled in advance, by you or whoever owns the product's quality, using the same rubric the judge uses. Run the judge on that set. Compute two numbers.

True positive rate. Of the outputs humans marked as failures, how many did the judge catch? If the judge misses half, the rubric is not operational. It doesn't matter that the judge could identify the failure mode in principle. On real outputs, with the rubric you wrote, it misses half.

True negative rate. Of the outputs humans marked as passing, how many did the judge correctly pass? If the judge flags a quarter of clean outputs as failures, a pipeline full of false alarms is a pipeline nobody reads.

TPR without TNR: we catch everything, including things that aren't broken. TNR without TPR: we agree on the easy cases and miss the hard ones. Together, the two numbers describe whether the judge is operating on the signal you thought the rubric pointed at.

When the numbers are bad, the fix is the rubric, not the judge.

This is the part people skip. A bad TPR looks like a judge problem — the judge is missing things, surely we need a smarter one. Switch to a bigger model. Turn up the temperature. Ask for a confidence score.

No. If the judge misses failures a human catches, the rubric isn't describing what the human is looking for. A rubric is a compression of human judgment; a compression that loses the main signal is a shadow of a rubric, not a rubric. Read the disagreements. A human marks an output as ungrounded because it invented a procedure name — the judge passed it because the rubric said "grounded in the customer's question" and the output did mention the customer's topic. "Grounded" is doing too much work. Name the specific failure: invents procedure names that do not appear in source documents. That's what a rubric looks like after the validation loop has pressured it.

Sometimes what the human was doing isn't rubric-able — they were using context the judge can't have, like knowledge of a recent incident or an unwritten product norm. That's worth knowing too. It tells you which failure modes belong in a human-in-the-loop stage, not inside a judge.

Calibration is continuous.

Models update. Prompts drift. Products change. A rubric with good TPR/TNR in March will score worse in June if nobody re-checks. The validation set is not a one-time artifact — it's a living corpus. Grow it. Re-label edges. Re-run the validation any time you change the judge model, the judge prompt, or the rubric. Treat a regression in validation numbers the same as a CI failure, because the judge is no longer reliable on the thing you claimed it was reliable on.

Without this, you have an eval system that produces numbers with no guarantee the numbers mean what they used to mean. Worse than no eval system, because the numbers are a confidence signal that causes people to stop looking.

This is the same shape as the reading is the work. Every part of the eval stack is downstream of humans reading outputs. The rubric is a compression of human reading. The judge is a model approximating the rubric. The validation set is humans re-grounding the approximation. Remove any of those and the whole thing starts measuring the wrong thing, confidently.

The judge is not a replacement for human reading. It's a cache. Caches need invalidation. The validation set is the invalidation signal — it tells you when the world has moved and the cache has to be rebuilt against current ground truth.

Binary judge scores. Hand-labeled validation set. TPR and TNR. Re-run when anything changes. Fix the rubric, not the judge.

Who validates the validators? You do. In a spreadsheet. By hand.