Deciding whether an LLM output is any good — rubrics, judges, offline datasets, and the gap between scoring well on a benchmark and working for your users.
14 posts
· 6 min read
A leaderboard measures the model; your product is the model plus your prompt plus your traffic. Nine criteria with how to measure each on your own inputs — and why the prompt you tuned for the incumbent is a rigged instrument for testing a challenger.
· 6 min read
Outcome decides, trajectory explains, cost constrains. Why the outcome assertion has to come from outside the agent, why a golden path punishes a better route, and why nine sampling steps mean more traffic rather than less.
· 5 min read
Temperature 0 makes decoding greedy, not deterministic — and the reasons are floating-point, batching, and routing. The reframe that fixes it: one output against one output is n = 1, and nobody accepts that anywhere else in engineering.
· 5 min read
The objection is fair, and weaker than it looks. Position bias, verbosity bias, self-preference, rubric drift — and the reframe that resolves the argument: a judge doesn’t have to be right, it has to be wrong the same way for both arms.
· 5 min read
No alert fires, because nothing breaks — the answers are still fluent, confident, and worse. Six causes with the detection method for each, and why a site-wide average is where regressions hide.
· 5 min read
Quality, operational, behavioural, business — four layers, what each answers, how fast it moves, and the trap in each. Plus why watching twenty metrics makes you less able to detect the effect you care about.
· 5 min read
Chunk size 512 because a blog post said so. Eight knobs, what each trades against, how to test it — and why retrieval metrics are diagnostic while end-to-end quality is the decision.
· 6 min read
“Hallucination rate” is four different failures wearing one word. The taxonomy with a detection method for each, why groundedness has to be scored per claim rather than per answer, and why the absolute number belongs nowhere near a sales deck.
· 5 min read
Ninety per cent of your scores are a 4 because an unanchored scale measures the judge’s prior, not your product. An anchored 1–5 table you can copy, six rules for the rubric itself, and why the rubric, the model, and the temperature version as one unit.
· 5 min read
“Valid” is five different claims, and the one that hurts is a well-formed object with an invented enum member falling through a switch with no default. Why truncation is usually the culprit, and why compliance is the cheapest guardrail you can declare.
· 5 min read
97% positive on a 1.5% response rate is a self-selected sample doing exactly what it should. Nine implicit signals with what each means and how each misleads, why a proxy needs validating once, and why behavioural metrics resolve before judge scores do.
· 5 min read
You can’t A/B test at forty users, which is not the same as being unable to measure. What each stage can and cannot tell you, the six things that work with no traffic, and why the plumbing is the expensive part rather than the statistics.
· 5 min read
Invented examples measure how you handle a user who read the docs; model-generated ones share your model’s blind spot by construction. The five strata with their shares, why the rubric matters more than the labels, and the questions your product can’t answer.
· 3 min read
Golden datasets are unit tests, not proof. Distribution drift, eval overfitting, and the metrics that only exist in production — latency, cost, and what users do next.