LLM A/B testing & prompt management
Run your prompt and model changes as real experiments on live traffic, and get a plain-English verdict on which one actually wins.
Open betaYoung product, fast fixes, direct line to the founders.
from llmjury import Client
client = Client(experiments=["checkout-prompt"])
llm = client.wrap(anthropic_client, "checkout-prompt")
# the two lines that turn your call into an experiment
p = client.get_prompt("checkout-prompt", user_id,
default="You are a helpful assistant.")
response = llm.messages.create(
model="claude-haiku-4-5",
system=p.prompt,
messages=[{"role": "user", "content": user_input}],
)That’s the whole integration — no stats code, no redeploy to change a prompt. The only metric you send by hand is your business outcome, via client.track(…). See the whole path in the five-minute quickstart.
The verdict, on a real experiment
Not a dashboard of numbers to interpret — a decision, in words: how many metrics moved, which way, and whether the split held. Quality, speed, cost, and business impact in one answer, with the full statistical detail one click away. SRM checks catch a broken split, and FDR correction keeps a lucky result from reading as a real one.
Every screenshot below is the real product, from the read-only sample experiment waiting in your account. It rotates on its own — use the arrows, or expand any shot to full screen for a closer look. Rather click through it? Open the interactive demo — same experiment, no sign-up.
Ten examples can look great while the change quietly makes half your real traffic worse.
Without measuring on your own traffic, you find regressions the way your users do — after they happen.
At a launch review, “I looked at some outputs” is the whole case. There is no number to point at.
Their history is `git blame`, changing a comma means a redeploy, and nobody can roll one back in a hurry.
LLMJury replaces “it feels better” with a measured answer from your own users — and moves your prompts out of the codebase into one place with full history.
Your prompt variants live in LLMJury, not your repo. Two lines of SDK put them behind one key — add a version without redeploying.
Each user is consistently assigned to a variant. LLMJury grades a sample of the outputs and tracks speed, cost, and conversions alongside.
“Variant B wins on quality, ties on speed, costs 12% less.” Promote it or revert — same screen, no deploy.
Design partners: 6 months of Business, free. We’re taking on a handful of teams shipping real LLM features — direct founder access and outsized say over the roadmap. 5 slots left.
Apply to be a design partner →Edit a prompt, save a new version, and deploy it without touching the repo or shipping a build. The engineer who owns the model call and the person who owns the words stop stepping on each other.
Every prompt keeps every version — no scrolling `git blame` to find what it said last month.
Who changed what, and when. Useful in a launch review; necessary the day something regresses.
Diff any two versions side by side, and revert to a known-good one instantly — no hotfix branch.
The SDK carries an in-code fallback prompt, so an LLMJury outage never takes your app down.
Worth signing up for this alone, before you run a single experiment. The version you edit is the variant you test is the verdict you ship.
Prompts live in LLMJury with version history, an audit trail, side-by-side compare, and one-click rollback. Deploy a new version without a redeploy — and the exact version you shipped is the one that gets tested.
Change the prompt, not the codebase. Prompts live in LLMJury with version history, an audit trail, side-by-side compare, and one-click rollback. Deploy a new version without a redeploy — and the exact version you shipped is the one that gets tested.
Test prompts and models on real traffic. Swap prompts, models, or parameters behind a variant key — no redeploys. The same user always sees the same variant, in every SDK, so the comparison is fair.
An LLM-as-judge scores your outputs for you. Built-in quality, safety, and relevance grading — plus your own rubric in plain English. Sampled and budget-capped, so grading never runs away with your bill.
Broken tests get stopped, not shipped. If your traffic split breaks, LLMJury halts the analysis instead of showing you a lie. Every result is checked against false positives before it reaches you. The full statistical detail is always one click away.
See if “better answers” actually moves the business. Send conversions or revenue events and read them next to quality scores — one experiment, one verdict, in the same place.
LLM-as-judge grading with SRM-gated, FDR-corrected statistics for defensible LLM evaluation.
Assignment is a local hash and the prompt comes out of client memory, so there is no client for us to support — LLMJury works with every provider, including ones we have never heard of. For OpenAI-shaped and Anthropic-shaped clients, one wrap() call also measures latency, tokens, and errors for you.
Every version in one place with an audit trail and instant rollback. Useful on day one, before you test anything.
How it works →Run two or more rewrites against each other on live traffic and get a graded verdict.
How it works →Route a slice of real traffic to the new model and read quality, speed, and cost side by side before you commit.
How it works →Temperature, max tokens, retrieval settings — test the whole permutation behind one variant key.
How it works →Ship every prompt change as an experiment first. A worse variant shows up in the dashboard, not a support ticket.
How it works →Not sure which fits? Schedule a demo and bring your use case.
Free gets you real experiments and full prompt versioning on 50k events a month. Upgrade for automatic quality grading and team seats; see full pricing.
$0/mo
Run your first real experiment and see the difference proof makes.
$149/mo
+$49 per extra 500k events
For teams shipping LLM features who need automatic quality grading.
$799/mo
+$99 per extra 2M events
For production-scale products and larger teams.
Custom
Custom limits, security review, SSO, and a dedicated contact.
LLMJury is built by PermuteLab LLC, out of years spent building observability platforms at massive scale. The statistical engine is open to inspection, the SDKs are open source on PyPI, npm, and Maven Central, and every question you send reaches a founder, not a queue — more about who is asking.
Don’t take the first one on trust — the changelog lists what shipped and when.
You’re sending us prompts and sampled model outputs, so here’s the plain version of what happens to them.
Your prompts, traffic, and sampled outputs run your experiments and nothing else. We never train models on customer data.
Raw events last 7 days on Free, 30 on Pro, 90 on Business; finalized results are kept. Email [email protected] to request deletion.
The wrap() path sends us your prompt and response text and we store it raw for your retention window. The key-only path sends none of it. Both are supported; pick the one your data allows.
Everything the SDK sends travels to our API over TLS.
The full picture — what the SDK sends and what it doesn’t, key scoping, sub-processors, and where we stand on certification — is on the security page.
We’re a young beta and haven’t been through formal certification (SOC 2, ISO) yet — ask us anything about our setup at [email protected] and we’ll answer straight.
Six months of Business free, founder access, and real influence on the roadmap. 5 slots left.
, wrap your model call, and your first experiment is live — free, no credit card.
Questions first? Email [email protected] or book a live demo— we answer everything ourselves, within one business day.