Evaluating LLM Applications in Python: Test Sets, Graders, Tracing, and Regression Reports
LLM evaluationPythonAI testingregression testingobservabilitydeveloper tools

Evaluating LLM Applications in Python: Test Sets, Graders, Tracing, and Regression Reports

UUpQbit Labs
2026-11-20
7 min read

Build a Python workflow for LLM evaluation with test sets, graders, traces, cost estimates, and CI-ready regression reports.

LLM applications can appear to improve while quietly becoming slower, more expensive, less reliable, or worse at retrieval and tool use. This practical Python workflow shows how to build versioned test sets, combine exact and rubric-based graders, capture traces, estimate token cost and latency, classify failures, and produce regression reports before a prompt, model, or pipeline change reaches production.

Overview

Evaluating an LLM application is different from checking whether a conventional function returns the expected value. A response may be factually correct but omit a required citation, use the wrong tool, expose unsupported information, or take too long to be useful. A strong evaluation process therefore measures several dimensions at once:

  • Answer quality: Is the response correct, relevant, complete, and appropriately formatted?
  • Grounding: Does the answer stay within the supplied documents or approved knowledge sources?
  • Tool behavior: Did the application call the correct tool with valid arguments, and did it avoid unnecessary calls?
  • Reliability: Does the same version behave consistently across representative cases?
  • Performance: How much time, input context, output, and tool activity does each request require?
  • Change impact: Did a new prompt, model, retrieval setting, or parser improve one category while damaging another?

The goal is not to reduce a nuanced application to one number. Instead, treat evaluation as a repeatable decision system. Every result should identify the application version, model configuration, prompt version, test case, trace, grader outputs, and failure category.

For structured responses, validate the output separately from its meaning. A schema check can catch missing fields or invalid types, while a quality grader can assess whether those fields contain a useful answer. The distinction is covered in more detail in Reliable JSON from LLM APIs in Python.

How to estimate

Start with a small, versioned evaluation set rather than an enormous collection of loosely defined examples. Store each case as data so it can be rerun after every meaningful change.

cases = [
    {
        "id": "billing-001",
        "input": "Can I change my billing date?",
        "expected_answer": "Explain the supported billing-date options.",
        "required_citations": ["billing-policy"],
        "expected_tools": ["search_docs"]
    },
    {
        "id": "billing-002",
        "input": "Delete my account and refund the last invoice.",
        "expected_answer": "Explain the account and refund process without claiming completion.",
        "required_citations": ["account-policy", "refund-policy"],
        "expected_tools": ["search_docs"]
    }
]

Use stable identifiers and add a case whenever a production failure reveals a missing scenario. Include normal requests, ambiguous requests, unsupported requests, long-context cases, formatting edge cases, and adversarial inputs relevant to your application.

For each run, record raw inputs and outputs along with metadata:

result = {
    "case_id": "billing-001",
    "app_version": "2026-03-12",
    "prompt_version": "support-v7",
    "model": "provider/model-name",
    "passed_exact": True,
    "quality_score": 0.8,
    "latency_ms": 1420,
    "input_tokens": 860,
    "output_tokens": 118,
    "tool_calls": ["search_docs"],
    "failure_category": None
}

Keep provider-specific metadata in a normalized internal format. Providers and SDKs may label usage, latency, finish reasons, or tool calls differently, so preserve the raw response for debugging and map only the fields your reports require.

Choose graders by question

Evaluation methodBest usePrimary limitation
Exact matchClass labels, fixed values, required keys, and deterministic routingPenalizes valid wording differences
Schema validationJSON shape, types, enumerations, and required fieldsDoes not prove semantic correctness
Reference comparisonShort answers with a clear expected resultRequires carefully maintained references
Rule-based checksCitations, prohibited claims, phrases, and tool argumentsCan miss meaning expressed differently
Rubric-based gradingRelevance, completeness, tone, groundedness, and reasoning qualityNeeds a stable rubric and calibration
Human reviewAmbiguous, high-impact, or newly discovered failure casesSlower and less consistent without review guidelines

A practical evaluation combines these methods. For example, use schema validation for output shape, exact checks for tool names, citation checks for document identifiers, and a rubric grader for answer quality. Do not allow a high semantic score to hide a failed safety or tool-use requirement.

Trace the complete request path, not only the final model call. A useful trace includes the user input, retrieved document IDs, prompt template version, model request, tool calls, intermediate results, final response, errors, timing, and token usage. Redact secrets and personal data before storing traces. The prompt engineering guide for developers provides related patterns for tool calling and structured output.

Inputs and assumptions

Before estimating quality, performance, or cost, define the inputs that can change the result. At minimum, record:

  • Dataset inputs: Number of cases, case categories, expected outputs, required citations, and expected tools.
  • Application inputs: Prompt version, system instructions, retrieval top-k, chunking settings, tool definitions, retry policy, and parser version.
  • Model inputs: Provider, model identifier, context settings, temperature or equivalent controls, and response limits.
  • Measurement inputs: Number of repetitions, timeout rules, grader versions, and whether failed requests count toward averages.
  • Cost inputs: Billable input units, output units, tool or retrieval charges where applicable, and fixed evaluation infrastructure costs.

Use formulas rather than hard-coded prices. If a provider publishes input and output rates, calculate an individual request as:

request_cost = (input_tokens / 1_000_000) * input_rate \
             + (output_tokens / 1_000_000) * output_rate \
             + tool_cost \
             + fixed_request_cost

Use the provider's current billing unit and rate card in your own configuration; do not embed assumptions that will become stale. For a test run, multiply the request estimate by the number of cases and repetitions, then add any separate retrieval, tracing, or infrastructure charges.

For quality, report both the overall pass rate and category-level results. A weighted score can help compare versions, but the weights must be explicit:

overall = (
    0.40 * groundedness_score
    + 0.30 * answer_score
    + 0.20 * tool_score
    + 0.10 * format_score
)

These weights are examples, not universal thresholds. A support assistant may prioritize groundedness, while a structured extraction service may prioritize schema validity. Define release gates such as “no critical tool-use failures” or “no more than a specified drop in groundedness,” then adjust them after reviewing real failure costs.

Worked examples

Comparing a prompt revision

Suppose version A and version B run against the same 40 cases with the same model and retrieval configuration. Version B produces better rubric scores but takes longer and makes more unnecessary tool calls. A useful report should show more than a single average:

MetricVersion AVersion BDecision question
Schema pass rate100%97.5%Did the revision damage integration reliability?
Groundedness score0.780.84Is the quality improvement meaningful?
Tool-call errors14Does the gain justify operational risk?
Median latency1,300 ms1,850 msDoes the user experience remain acceptable?
Estimated run costConfigured rateConfigured rateIs the change affordable at expected volume?

Investigate the individual cases behind every regression. A lower overall score may be acceptable if it comes from low-priority cases, while one failure in a critical workflow may block release.

Estimating evaluation cost

If a suite contains 120 cases and each case is run three times, the evaluation performs 360 application runs. If each run can invoke a model grader, include those grader calls in the estimate separately. Retrieval and tool calls may also generate their own usage. Keep a ledger with one row per call type:

total_runs = cases * repetitions
application_calls = total_runs
quality_grader_calls = total_runs
cost_estimate = (
    application_calls * application_cost
    + quality_grader_calls * grader_cost
    + retrieval_cost
    + tool_cost
)

When rates are unknown or variable, report the measured token totals and call counts instead of inventing a currency estimate. Those measurements can be multiplied by the current provider inputs during a later review.

When to recalculate

Re-run the evaluation suite before deploying a prompt, model, retrieval, tool, parser, or dependency change. Recalculate cost and performance whenever pricing inputs, tokenization behavior, context settings, rate limits, or traffic assumptions change.

Also revisit the suite on a regular schedule. A quarterly benchmark review is a reasonable operating rhythm for many teams, with additional reviews after major model or framework releases. Refresh model metadata, SDK response fields, tracing adapters, evaluation-library APIs, and pricing configuration rather than assuming earlier integrations still report the same values.

Use production evidence to expand the test set. Categorize failures as retrieval miss, unsupported claim, incomplete answer, formatting error, wrong tool, invalid tool arguments, timeout, rate-limit failure, unsafe response, or grader disagreement. Add a minimized reproduction for important failures and tag it with severity and source. This turns incidents into permanent regression protection.

For a CI-friendly workflow, fail the build only on explicit release gates: critical policy violations, invalid output schemas, unacceptable tool errors, or a quality decline beyond an agreed tolerance. Store full traces and reports as artifacts, but keep secrets out of logs. Review trends across versions rather than treating one run as definitive, especially when model outputs are nondeterministic.

Finally, keep the evaluation implementation itself tested. Mock provider responses for parser tests, use recorded traces for grader tests, and run a small deterministic smoke suite on every change. For broader Python testing patterns, see Testing Circuits in Python with Pytest; the same separation between fast checks and slower integration runs applies to AI applications. A repeatable test set, transparent assumptions, and trace-backed reports give developers a defensible answer to the central question: did the application improve, and what did that improvement cost?

Related Topics

#LLM evaluation#Python#AI testing#regression testing#observability#developer tools
U

UpQbit Labs

Editorial Team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.