LLMJury Blog Run live A/B tests on your prompts and models, version prompts outside your codebase, and get a plain-English verdict backed by real statistics — LLM evaluation with proof, not gut feeling. https://llmjury.com/blog 2026-08-07T12:00:00.000Z LLMJury How to choose an LLM for your product https://llmjury.com/blog/how-to-choose-an-llm 2026-08-07T12:00:00.000Z 2026-08-07T12:00:00.000Z A leaderboard measures the model; your product is the model plus your prompt plus your traffic. Nine criteria with how to measure each on your own inputs — and why the prompt you tuned for the incumbent is a rigged instrument for testing a challenger. How do you evaluate an AI agent? https://llmjury.com/blog/evaluating-ai-agents 2026-08-06T12:00:00.000Z 2026-08-06T12:00:00.000Z Outcome decides, trajectory explains, cost constrains. Why the outcome assertion has to come from outside the agent, why a golden path punishes a better route, and why nine sampling steps mean more traffic rather than less. Same prompt, different answer: testing a model that won’t sit still https://llmjury.com/blog/llm-non-determinism-testing 2026-08-05T12:00:00.000Z 2026-08-05T12:00:00.000Z Temperature 0 makes decoding greedy, not deterministic — and the reasons are floating-point, batching, and routing. The reframe that fixes it: one output against one output is n = 1, and nobody accepts that anywhere else in engineering. Can you trust an LLM to grade an LLM? https://llmjury.com/blog/can-you-trust-llm-as-judge 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z The objection is fair, and weaker than it looks. Position bias, verbosity bias, self-preference, rubric drift — and the reframe that resolves the argument: a judge doesn’t have to be right, it has to be wrong the same way for both arms. How much traffic do you need to A/B test a prompt? https://llmjury.com/blog/how-much-traffic-do-you-need 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z “We’re too small to A/B test” is usually wrong. A sample-size table for the metrics teams actually run — task success, judge quality, cost, latency — and the four levers that make small traffic workable. Get your prompts out of your codebase https://llmjury.com/blog/prompt-versioning-guide 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z A hardcoded prompt means a deploy to change a comma, no real rollback, no history — and, invisibly, no way to run two versions at once. The five properties a prompt version actually needs. Your experiment is lying to you: sample ratio mismatch https://llmjury.com/blog/sample-ratio-mismatch 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z A 3% imbalance on 100,000 users is not noise, it is a bug — and it invalidates every number in the experiment, not just the split. What causes SRM, why a timeout is the LLM-specific culprit, and why you must never reweight. Upgrading to a new model without breaking your product https://llmjury.com/blog/model-upgrade-without-regressions 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z The new model benchmarks better and performs worse, because your prompt is a fossil record of the old model’s flaws. A five-step migration sequence with exit criteria, and why you should leave a comparison arm running forever. Cutting LLM costs without cutting quality https://llmjury.com/blog/cheaper-model-same-quality 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z Four levers ranked by effort, and the statistical point that decides it: you are not asking whether the cheap model is better, you are asking whether it is worse by less than you will trade for the money. That is a non-inferiority test. When can you call it? Peeking, p-hacking, and stopping an experiment https://llmjury.com/blog/when-can-you-call-it 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z Fourteen daily looks at a p-value turns a 5% error rate into roughly one in five — and inflates every effect you report. Stopping rules that a real team will actually follow, plus the five lines to write down before you start. The silent regression: why LLM quality drops and nobody notices https://llmjury.com/blog/the-silent-regression 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z No alert fires, because nothing breaks — the answers are still fluent, confident, and worse. Six causes with the detection method for each, and why a site-wide average is where regressions hide. What to actually measure in an LLM product https://llmjury.com/blog/what-to-measure-llm-product 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z Quality, operational, behavioural, business — four layers, what each answers, how fast it moves, and the trap in each. Plus why watching twenty metrics makes you less able to detect the effect you care about. A/B testing your RAG pipeline https://llmjury.com/blog/ab-testing-rag 2026-08-04T12:00:00.000Z 2026-08-04T12:00:00.000Z Chunk size 512 because a blog post said so. Eight knobs, what each trades against, how to test it — and why retrieval metrics are diagnostic while end-to-end quality is the decision. How to put a number on hallucination https://llmjury.com/blog/measuring-hallucination-rate 2026-08-03T12:00:00.000Z 2026-08-03T12:00:00.000Z “Hallucination rate” is four different failures wearing one word. The taxonomy with a detection method for each, why groundedness has to be scored per claim rather than per answer, and why the absolute number belongs nowhere near a sales deck. Fine-tuning or prompting: which problem do you actually have? https://llmjury.com/blog/fine-tuning-vs-prompting 2026-08-02T12:00:00.000Z 2026-08-02T12:00:00.000Z “The prompt isn’t working” is four problems wearing one sentence, and fine-tuning fixes one of them. The symptom-to-technique table, the four costs that land after the training run, and why most published fine-tuning wins are measured against a first-draft prompt. How to write a judge rubric that means the same thing in six months https://llmjury.com/blog/writing-judge-rubrics 2026-08-01T12:00:00.000Z 2026-08-01T12:00:00.000Z Ninety per cent of your scores are a 4 because an unanchored scale measures the judge’s prior, not your product. An anchored 1–5 table you can copy, six rules for the rubric itself, and why the rubric, the model, and the temperature version as one unit. A/B testing a chatbot: what counts as one observation? https://llmjury.com/blog/ab-testing-chatbots-multi-turn 2026-07-31T12:00:00.000Z 2026-07-31T12:00:00.000Z Sixty thousand turns is not sixty thousand observations. The unit-of-analysis error doesn’t make an experiment noisy, it makes it overconfident — plus which metrics live at turn, conversation, and user level, and why turns-to-resolution lies on its own. Your experiment came back flat. Now what? https://llmjury.com/blog/null-result-playbook 2026-07-30T12:00:00.000Z 2026-07-30T12:00:00.000Z “No significant difference” is three different results with the same p-value, and only the confidence interval tells them apart. What a null actually buys you, the four things not to do next, and why most flat results are a design problem rather than a traffic problem. When the JSON stops parsing: structured output as a metric https://llmjury.com/blog/structured-output-reliability 2026-07-29T12:00:00.000Z 2026-07-29T12:00:00.000Z “Valid” is five different claims, and the one that hurts is a well-formed object with an invented enum member falling through a switch with no default. Why truncation is usually the culprit, and why compliance is the cheapest guardrail you can declare. The average is hiding the answer: segmenting LLM experiments https://llmjury.com/blog/segmenting-llm-experiments 2026-07-28T12:00:00.000Z 2026-07-28T12:00:00.000Z Almost every prompt change trades one segment against another, so a flat average is a suspicious result rather than a boring one. A worked Simpson’s paradox where the pooled split passes an SRM check, the segments worth declaring, and why a split result is a design brief. LLM latency: the number your users actually feel https://llmjury.com/blog/llm-latency-what-users-feel 2026-07-27T12:00:00.000Z 2026-07-27T12:00:00.000Z TTFT, tokens per second, total completion, end-to-end — four numbers that diverge, and your interface decides which one matters. Why the tail is fat, why output length is the real lever, and why a latency gap between arms is usually an SRM. Thumbs up is not a metric: reading what users actually tell you https://llmjury.com/blog/user-feedback-signals 2026-07-26T12:00:00.000Z 2026-07-26T12:00:00.000Z 97% positive on a 1.5% response rate is a self-selected sample doing exactly what it should. Nine implicit signals with what each means and how each misleads, why a proxy needs validating once, and why behavioural metrics resolve before judge scores do. Measuring what your LLM does under attack https://llmjury.com/blog/prompt-injection-monitoring 2026-07-25T12:00:00.000Z 2026-07-25T12:00:00.000Z Injection has no clean fix, so the answerable question is what fraction gets through and whether that moved. Blocked, deflected, complied, over-refused — and why the outcome nobody counts is the one that quietly costs you real users. Bandits or A/B tests for LLM features? https://llmjury.com/blog/bandits-vs-ab-testing 2026-07-24T12:00:00.000Z 2026-07-24T12:00:00.000Z A bandit optimises, an experiment explains — and you get one. The comparison table, plus the reason bandits fit LLM products badly: they need fast feedback, and a judge score sampled at 5% on a batch is not fast feedback. Can you run two LLM experiments at the same time? https://llmjury.com/blog/multiple-experiments-at-once 2026-07-23T12:00:00.000Z 2026-07-23T12:00:00.000Z Independent randomisation makes a concurrent experiment noise rather than bias — so the default answer is yes, and queueing has a cost nobody puts on a dashboard. The four cases where it breaks, and the two-line salt bug that silently correlates every arm. Prompt caching: the discount most teams leave on the table https://llmjury.com/blog/prompt-caching-cost 2026-07-22T12:00:00.000Z 2026-07-22T12:00:00.000Z The cache breaks at the first token that differs, and everything below it is recomputed at full price — so one timestamp near the top of a 3,000-token system prompt voids the whole discount. Order by volatility, and track hit rate per variant. How to test an LLM feature before you have users https://llmjury.com/blog/testing-before-you-have-users 2026-07-21T12:00:00.000Z 2026-07-21T12:00:00.000Z You can’t A/B test at forty users, which is not the same as being unable to measure. What each stage can and cannot tell you, the six things that work with no traffic, and why the plumbing is the expensive part rather than the statistics. Build your eval set out of production traffic https://llmjury.com/blog/eval-set-from-production 2026-07-20T12:00:00.000Z 2026-07-20T12:00:00.000Z Invented examples measure how you handle a user who read the docs; model-generated ones share your model’s blind spot by construction. The five strata with their shares, why the rubric matters more than the labels, and the questions your product can’t answer. Writing up an experiment so the decision actually sticks https://llmjury.com/blog/reporting-experiment-results 2026-07-19T12:00:00.000Z 2026-07-19T12:00:00.000Z An experiment’s output is a decision plus the record that makes it re-examinable — otherwise the losing variant gets re-proposed in six weeks. A seven-section template, how to state uncertainty to people who won’t hear “significant” correctly, and the incentive check. Logging LLM traffic without creating a privacy incident https://llmjury.com/blog/logging-llm-traffic-privacy 2026-07-18T12:00:00.000Z 2026-07-18T12:00:00.000Z The numbers an experiment needs are not the text. The minimum viable event, field by field, with what is personal data and what isn’t — plus why a pseudonymous id is still in scope, and why text and metrics need completely different retention. Why prompt changes deserve A/B tests, not vibes https://llmjury.com/blog/why-ab-test-prompts 2026-07-16T12:00:00.000Z 2026-07-16T12:00:00.000Z Ten playground outputs can’t tell you what a prompt change does to real traffic. The case for measuring every prompt edit the way you measure every other production change. Offline evals aren’t enough: why prompts need testing on live traffic https://llmjury.com/blog/offline-evals-are-not-enough 2026-07-16T12:00:00.000Z 2026-07-16T12:00:00.000Z Golden datasets are unit tests, not proof. Distribution drift, eval overfitting, and the metrics that only exist in production — latency, cost, and what users do next. Announcing LLMJury: statistically defensible A/B testing for LLM products https://llmjury.com/blog/announcing-llmjury 2026-07-01T12:00:00.000Z 2026-07-01T12:00:00.000Z Why “it feels better” is not an eval strategy, and how SRM gates, FDR correction, and an LLM judge make prompt experiments trustworthy.