Write the eval set first
The single highest-leverage thing you can do on an LLM project is write the eval set on day one. Not day thirty. Day one. A hundred labelled input/output pairs, sourced from real user requests where possible, is worth more than any prompt engineering that comes after.
If you cannot write the eval set — if you cannot describe what 'right' looks like — you do not have a feature yet. You have a research project. That's fine; just don't ship it as a feature.
Offline evals: regression harness
Every prompt change, model change, retriever change and tool change runs against the eval set in CI. The output is a score per rubric (accuracy, safety, format, tone) and a diff against the previous baseline. A drop of more than 2% on any rubric blocks the merge.
The rubric prompts are versioned in the repo. The judge model is pinned. You cannot debug a moving eval.
Online guardrails: the runtime layer
Offline evals cannot cover the long tail of real inputs. Online guardrails do. Ship at least four:
- Input classifier — reject or route obviously out-of-scope requests before they hit the model.
- Output validator — enforce the schema or format the downstream system expects. Never trust the model to be a parser.
- PII / secrets scrub — on both input and output. Log the scrubbed version, never the raw.
- Rate & cost limits — per user, per tenant, per feature. The runaway is not hypothetical.
Staged rollout: 1% → 10% → 50% → 100%
Every LLM feature ships behind a flag. First cohort is internal staff. Then 1% of external users for at least three days. Then 10% for a week. Only then does the ramp continue. At each stage, three signals must be green: the eval set on production traffic, the guardrail trip rate, and the outcome metric the feature was built to move.
Killswitch: test it, don't just build it
Every LLM feature has a killswitch. A boolean flag that instantly reverts to a deterministic fallback — often 'don't show the feature' or 'show the old, non-LLM version'. The killswitch is tested in staging, in production, and in a chaos drill once a month. A killswitch that has never been pulled is not a killswitch, it's a hope.
The metrics that matter
Outcome metrics (deflection rate, task completion, revenue) are what you're paid to move. Leading indicators are what tell you a week early. Track both.
- Refusal rate — a sudden spike usually means a model update or a prompt regression.
- Tool-error rate — the model is calling tools wrong, or the tools have drifted.
- Median and p95 latency per feature — LLM latency is bimodal; the mean lies.
- Cost per successful outcome — not cost per call. This is the number your CFO cares about.
- Human-override rate — where a human corrects the model. This is your gold-labelled eval data, for free.
The weekly review nobody wants to book
Book a 45-minute weekly review with engineering, product and one domain expert. Walk through: eval scores this week vs last, guardrail trips, top 10 human overrides, cost trend, one interesting failure. This meeting is where you catch drift before it becomes an incident. Skipping it saves 45 minutes a week and costs you a quarter.
FAQ
- How big should my eval set be?
- Start with 50 hand-picked cases covering happy path, edge cases and known failure modes. Grow to 200–500 as real user data accumulates. Beyond that, precision matters more than volume — a small, well-labelled set beats a large, noisy one.
- LLM-as-judge or human eval?
- Both. LLM-as-judge for every commit — it's cheap and consistent. Human eval on a sampled slice every week — it catches judge model drift and blind spots.
- How do we handle a model provider upgrade?
- Treat it like any other dependency bump. Run the full eval set against the new model behind a flag. Ramp with the same 1/10/50/100 curve. Never take a silent upgrade on a feature that touches customers.
Building something like this?
Simatech ships AI-native Scrum squads for teams doing exactly this work — MCP agents, GraphRAG, LLMOps and the rest.
Talk to us