02/ 03Case study
AIRE: AI Reliability & Evaluation Engine
A trace-based evaluation engine that turns LLM/RAG behavior into replayable evidence: deterministic metrics, a failure taxonomy, and direction-aware regression detection.
- Python
- pytest
- HTML/CSS
Lab/See it run
QueryA fixed 26-case JSONL suite goes in - 22 answerable, 4 unanswerable. Every run manifest stores the dataset's SHA-256.
01/Overview
What this is.
AIRE runs a system under test once over a fixed JSONL suite, persists schema-validated per-invocation traces, and evaluates the recorded traces with deterministic metrics - never re-running the system. Runs are compared with a direction-aware, relative-threshold regression engine that attaches per-case attribution to every regression, and every failure lands in a mutually exclusive taxonomy.
The framework is zero-dependency and fully reproducible offline: a deterministic RAG stack (keyword retrieval + extractive responder) provides the calibration harness, and a separate recorded run evaluates real Gemini generation against the same suite. Both live in the same trace format, so the comparison is apples-to-apples.
02/Problem
What motivated it.
AI systems are easy to build and hard to trust. Evaluation is nondeterministic, reference answers are scarce, and every LLM call costs money - so most testing happens once, informally, at build time, and never again.
The gap this project attacks: make evaluation an artifact. Run once, store the evidence, and let metrics, failure analysis and regression detection be re-run forever against stored traces - including on runs collected months earlier from a different system.
03/System
How it is built.
- runner/dataset.py
- EvalDataset: the fixed JSONL suite. Every run manifest embeds the dataset's SHA-256, so traces are attributable to an exact suite version.
- llm/base.py + deterministic.py + openai_compat.py
- The SystemUnderTest interface with two implementations: a deterministic stack (KeywordRetriever + ExtractiveRAG with faithful/fluent styles) and a real LLM via an OpenAI-compatible endpoint.
- runner/runner.py
- ExperimentRunner: per-invocation timeout watchdog (30 s default, 150 s for LLM runs), retries, and error propagation - SUT failures become trace errors, never silent empty answers.
- tracing/
- TraceRecorder + RunStore: schema-validated traces persisted as manifest.json + traces.jsonl per run directory.
- evaluator/engine.py
- EvaluationEngine: applies metrics and the first-match-wins failure taxonomy to stored traces. It never re-runs the system under test.
- regression/compare.py
- Direction-aware run comparison with relative thresholds and per-case attribution, sorted worst-first.
- report/html.py
- Static HTML comparison reports generated from the stored runs.
04/Approach
The mechanisms that matter.
Run once, replay forever
Traces carry the dataset hash and full invocation records, so new metrics apply to old runs at zero API cost and comparisons never re-invoke the system.
Deterministic metrics, defined in code
Keyword recall, citation coverage, a lexical groundedness proxy (at least 50% content-keyword overlap with the cited document), abstention correctness, and recall@k/MRR - each metric's blind spots documented in its docstring rather than oversold.
Direction-aware regression
A metric registry declares higher/lower-is-better per metric; thresholds are relative to the baseline (2% default) so one setting spans metrics of wildly different scales, with a looser override for latency.
First-match-wins failure taxonomy
Invocation error, then abstention errors, then ungrounded, incomplete, retrieval miss, success. The ordering encodes causality, so each case lands in exactly one bucket and counts sum to the suite.
Honest operational data
Timeouts, retries, token provenance (api vs heuristic) and rate-limit failures all land in traces as invocation_error records - reliability is measured, not excused.
05/Evaluation
Setup and workloads.
- Suite: 26 JSONL cases over a 12-document corpus - 22 answerable, 4 unanswerable (expecting abstention), 2 requiring multi-doc synthesis.
- Three deterministic RAG versions: v1 overlap retrieval top-1 with faithful quoting; v2 IDF-weighted top-3 retrieval; v3 = v2 plus a fluent style that merges sentences and drops citations - a planted regression.
- Live run: gemini-2.5-flash through an OpenAI-compatible endpoint, temperature 0, local IDF top-3 retrieval, 5 s pacing between calls, 150 s runner timeout. The API key is read from the environment and never written to traces.
06/Results
Measured outcomes.
- 0.57→0.00
- groundedness collapse caught (planted v3 regression)
- 0.91
- retrieval recall after v2 (v1: 0.73)
- 1.00
- abstention + grounding on the live run (0.0 / 0.57 before)
- 17/26
- clean cases on live Gemini (6 quota errors recorded)
| Metric | v1 baseline | v2 improved | v3 fluent | Live Gemini |
|---|---|---|---|---|
| Retrieval recall | 0.7273 | 0.9091 | 0.9091 | 0.9091 |
| Citation coverage | 0.7273 | 0.7273 | 0.0 | 0.6818 |
| Groundedness | 0.5682 | 0.5682 | 0.0 | 1.0 |
| Abstention correct | 0.0 | 0.0 | 0.0 | 1.0 |
| Latency / case | 0.22 ms | 0.22 ms | 0.36 ms | 15 320 ms |
| Clean cases | 12/26 | 13/26 | 0/26 | 17/26 |
All values from results/*/eval_results.json, llm_eval_results.json and the two comparison JSONs. The live run recorded 6/26 cases as invocation_error traces (HTTP 429 quota errors), one wrong abstention on an answerable case, and exact API token totals (12 374 prompt + 756 completion).
07/Findings
What the results say.
- 01
Fluency destroyed grounding: v3 read better and kept v2's retrieval gain (recall 0.73 -> 0.91) while every citing case became ungrounded (groundedness 0.57→0.00, citation coverage 0.73→0.0). A composite score would have called v3 fine.
- 02
Better is a trade-off table, not a number: the live responder fixed abstention (0.0→1.0) and grounding (0.57→1.0) but cost 15.3 s vs 0.3 ms per case and 7.3x the tokens.
- 03
The evaluator exposed a capability gap: unanswerable retrieval scores (0.18-0.46) overlap answerable minimums (0.19-0.26), so no score threshold can fix abstention - it needs semantic matching.
- 04
Operational reliability is measured, not excused: 6/26 quota failures are recorded as invocation_error traces, and 17/26 clean is reported as-is.
- 05
Groundedness is a documented proxy: it catches uncited claims but cannot certify entailment - the README states the blind spot instead of overselling.
08/Decisions
Why it was built this way.
Replay economics
Run once, store traces, evaluate forever: new metrics apply to old runs at zero API cost, and comparisons never re-invoke the system.
A deterministic SUT exists
Every committed number is reproducible offline, and evaluator bugs surface as metric changes instead of LLM noise.
Relative thresholds
Delta over baseline makes the default 2% scale-free across metrics of wildly different ranges: +0.58 tokens on ~83 is noise; -0.57 groundedness is a catastrophe.
No evaluation framework
Not building on DeepEval/Ragas keeps every metric auditable line-by-line - the evaluation layer is the product.
Errors propagate
SUT-internal failures surface as trace errors and classify as invocation_error, so provider failures cannot masquerade as wrong answers.
09/Validation
How the claims are checked.
- A 20-test suite covers metric definitions (supported/unsupported/uncited sentences), recall@k and MRR, trace schema validation and round-trip, RunStore rejection of invalid traces, retriever ranking, faithful-cites/fluent-drops behavior, taxonomy priority, regression direction on the fluent run, and HTML report rendering.
- Three real bugs found by the harness are documented in-repo: dropped SUT errors misclassifying provider failures, a 30 s watchdog that consumed 14/26 live cases (raised to 150 s), and a citation-attachment parsing bug.
- Dataset integrity: every run manifest embeds the SHA-256 of eval_set.jsonl, so results are attributable to an exact suite version.
10/Limitations
Stated plainly.
- Lexical groundedness is a proxy, not entailment: faithful paraphrase scores zero and vocabulary reuse can fool it.
- The deterministic responder is a reproducible harness, not a real LLM; the live Gemini run is a 26-case single sweep with 6 quota failures - a demonstration, not a leaderboard.
- Keyword retrieval cannot decide answerability (the measured score distributions overlap); abstention remains an open gap for the deterministic stack.
- Token counts for the deterministic runs are heuristic (about 4 chars/token), and the thread-based watchdog cannot kill hung workers.
- Retrieval ground truth is hand-declared.
11/Interview TL;DR
- Built
- A trace-based evaluation engine for LLM/RAG systems: one 26-case suite, three deterministic RAG versions, a live Gemini run, and a regression engine with per-case attribution.
- Why
- AI systems are easy to build and hard to trust - nondeterminism, scarce references and call cost make point-in-time testing worthless. AIRE makes evaluation replayable evidence.
- How it works
- System runs persist as schema-validated traces; a deterministic engine scores them, classifies failures first-match-wins, and compares runs with direction-aware relative thresholds.
- Key decision
- Run once, replay forever: traces carry a dataset hash, so new metrics apply to old runs at zero API cost.
- Strongest result
- The planted v3 fluent regression: retrieval held at v2's 0.91 while groundedness fell 0.57→0.00 - caught, and attributed to 22 specific cases (17 for citation coverage).
- Main limitation
- Groundedness is a lexical proxy, and the live LLM run is a small-n single sweep (17/26 clean, 6 quota errors).
- Next
- A semantic-entailment second-tier judge, and a score-free abstention path, since the measured score distributions overlap.
Likely questions
Q01Why replay traces instead of evaluating live?
Cost and determinism: the SUT runs once and the engine scores stored traces, so new metrics apply to old runs at zero API cost and comparisons never re-invoke the system.
Q02What does your groundedness metric actually measure?
A lexical proxy: per cited sentence, at least 50% content-keyword overlap with the cited document. It catches uncited claims but is not entailment - faithful paraphrase scores zero, and the README documents that.
Q03How does the regression engine avoid crying wolf?
Thresholds are relative to the baseline (2% default), direction-aware per metric, with a looser latency override; every regression ships with per-case attribution sorted worst-first.
Q04Why a first-match-wins failure taxonomy?
It encodes causality - an invocation error cannot also be an ungrounded answer - so each case lands in exactly one bucket and the counts sum to the suite.
Q05Why not use DeepEval or Ragas?
The evaluation layer is the product. I want every metric auditable line-by-line, and the deterministic mode lets evaluator bugs surface as metric changes instead of LLM noise.