Evaluating LLM applications before they reach users
Without evaluation, every prompt or model change is a guess. How to build an evaluation practice for LLM applications that fits into normal software delivery.
- By
- Plenware Engineering
- Published
- Reading time
- 2 min read
Table of contents
Traditional software has a comfortable property: given the same input, it produces the same output, and a test can check it. LLM applications do not behave that way. Outputs vary in wording, quality depends on subtle prompt and model choices, and a change that improves one case can quietly break another.
Teams that ship reliable LLM features treat evaluation as part of engineering, not a final review. Here is a practical way to build that practice.
Define what good looks like
Start by writing down, with domain experts, what a good response means for your application. For a support assistant it may be correct, grounded in the knowledge base, polite, and concise. For an extraction pipeline it may be exact field values in a valid schema.
These criteria become the dimensions you score.
Build a test set from real cases
Collect a few hundred representative inputs, more over time, covering:
- Common requests
- Difficult and ambiguous requests
- Inputs that should be refused or escalated
- Known past failures
Where possible, attach a reference answer or the facts a good answer must contain. Store the set in version control next to the code.
Score with the right method for each dimension
| Dimension | Method |
|---|---|
| Structured output | Deterministic checks: schema validation, exact match |
| Factual content | Check for required facts, or compare to reference answers |
| Groundedness | Verify claims against retrieved sources |
| Tone and helpfulness | Model based grading calibrated against human ratings |
Model based grading is useful at scale, but calibrate it. Have people rate a sample, compare their scores with the grader's, and refine the grading instructions until they agree.
Run evaluations in CI
Treat prompts, model versions, and retrieval settings like code. When a pull request changes any of them, run the evaluation and report the results alongside unit tests. Set thresholds for critical dimensions so regressions block a merge.
Full evaluations can be slow and costly, so many teams run a smaller smoke set on every change and the full set nightly or before release.
Track cost and latency too
A change that improves quality by a point but doubles latency may be the wrong trade. Record tokens, cost per request, and response time for every evaluation run.
Close the loop with production
Pre release evaluation only covers what you anticipated. In production, capture user feedback, flag low confidence responses, and sample conversations for review, with privacy controls in place. Add new failure cases to the test set. Over time, the test set becomes a precise description of what your application must handle.
Start small
You do not need a platform to begin. A spreadsheet of fifty cases, a script that runs them, and a habit of checking results before every release will catch most regressions. Add automation and tooling as the application and the team grow.
Plenware builds evaluation into every AI engineering engagement, so quality is measured from the first prototype onward.
- LLM
- Evaluation
- Testing
- MLOps