How to evaluate LLMs before production

Defining the Product Decision, Not the Model

When an LLM system doesn’t perform as expected, the first instinct is often to adjust its technical components. Teams may rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models. Before making any of these changes, they should define the decision the evaluation is meant to support.

For our secret-scanning work, we asked: Can the system reduce false positives while preserving enough recall to be safe in a production security workflow? To answer this question, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds.

In secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert. We therefore did not treat precision and recall as equally interchangeable metrics. Our primary objective was to reduce false positives and improve precision. Recall served as a safety constraint: an experiment could advance only if any decrease remained within a predefined acceptable range.

Treating Offline Evaluation Like Integration Testing

An LLM-based system continues to change after its first successful evaluation, so evaluation should not be a one-time exercise. Teams revise prompts, adopt new models, change how inputs and context are constructed, and refine the surrounding business logic. Any of these changes can improve the system, introduce a regression, or shift its behavior in an unexpected way.

For that reason, we treated offline evaluation similarly to an end-to-end integration test. We reran it whenever we made a meaningful change to the prompt, model, input construction, or broader system logic. The evaluation also needed to be repeatable enough that each new result could be compared against a known baseline.

Organizing Evaluation Criteria

We organized the evaluation criteria into three levels: primary outcome, safety constraint, and operational guardrails. This distinction prevented us from treating every metric as interchangeable. A change that reduced false positives but significantly lowered recall wasn’t automatically an improvement. Neither was a change that improved quality while making the system too slow, expensive, or difficult to integrate.

Consider two hypothetical experiment results:

Experiment Precision Recall Latency Decision Experiment A Large improvement Falls below the safety guardrail Acceptable Don’t advance Experiment B Moderate improvement Remains within the guardrail Acceptable Continue testing

Experiment A may look stronger if precision is viewed in isolation. Experiment B is more aligned with the product goal because it improves the developer experience without violating the recall guardrail.

Conclusion

Evaluating LLMs before production is a challenging task. By defining the product decision, not the model, and treating offline evaluation like integration testing, teams can ensure that their LLM-based system is safe and effective in a production environment.