AIRE · AI Reliability & Evaluation Engine

Record, evaluate, compare - and catch what a pass/fail number hides

AIRE is a trace-based evaluation framework for LLM/RAG systems. Every invocation is recorded as a validated trace; deterministic metrics and a failure taxonomy score the records; a regression engine compares versions metric-by-metric with per-case attribution. This explorer renders the committed evaluation of a small RAG system - three deterministic versions plus one recorded live-LLM run.

What the evaluator caught in 60 seconds: version 3 reads better and retrieves better (recall 0.73 → 0.91) but silently drops every citation - groundedness 0.57 → 0.00 across 22 ungrounded cases, flagged automatically as a regression. The live-LLM run then fixes abstention (0.0 → 1.0) and grounding (1.0) at the price of 15 s/case and ~605 tokens/case. No single number tells this story; the metric set does.

Explore a run

Aggregate metrics first, then per-case evidence. Evaluation reads traces - the system is never re-run. measured result

Watch v1 become v2 become v3

The committed metrics, animated between the three deterministic versions. Watch retrieval climb while groundedness holds - then collapses the moment the fluent style lands. measured result

Retrieval recall (higher is better)

Groundedness (higher is better)

Replay one case through the evaluator

The evaluator's decision path for a single recorded case: run → trace → retrieval → answer → citation check → groundedness → failure taxonomy. Every value shown is from the stored trace and committed metrics. measured result

Regression comparisons

Direction-aware relative thresholds (higher-better vs lower-better), with every regression attributed to the cases that moved. Verdicts come from the committed comparison reports. measured result