LLM Evaluation Harness
THE PROBLEM
At Lid Vizion, prompt tweaks and model swaps were shipping on anecdote — someone eyeballed a few outputs, said "looks better," and users found the regressions. Teams that cannot measure model quality cannot ship it safely, and we could not measure it. I own the harness that turned quality into a CI gate, including the uncomfortable discovery that our LLM-judge metrics had drifted from human judgment and were green-lighting releases reviewers later rejected.
CONSTRAINTS
- Model providers change behavior without warning
- Quality, cost, and latency trade against each other
- LLM-judge metrics drift from human judgment — we got burned by it
- Rollbacks must be fast when a release regresses
DECISIONS & TRADEOFFS
GOLDEN DATASETS over ad-hoc spot checks — because a fixed labeled set makes every prompt and model version comparable against the same ground truth
CI REGRESSION GATES over manual review — because quality checks that run on every change catch regressions before merge, not after deploy
VERSIONED PROMPTS AND MODELS over config drift — because every release is pinned and reproducible, so rollback is a pointer move, not archaeology
ARCHITECTURE
OUTCOME
Prompt and model changes stopped being leaps of faith. Every release carries quality, cost, and latency readings against the same golden sets, regressions fail in CI instead of arriving as user complaints, and judge metrics get recalibrated against human review on a schedule. A bad deploy rolls back in minutes because every version is pinned.
PROOF / LINKS
AT 100× SCALE
At 100x this becomes the quality control plane for every LLM feature in the org: per-product golden sets, shadow evals on production traffic, and automatic rollback when live metrics drift from the gate.