Evals & benchmarks

The proof-of-value benchmark

The question the benchmark answers: does retrieval actually change what a coding agent ships?

Setup

• 12 React + TypeScript tasks, each with required component APIs expressed as regex patterns.
• Corpus: @halcyon/ui 3.4.0 — a fictional design system (38 files, 93 chunks) invented so no frontier model can have memorized it. A high bare score is impossible to explain away as training data.
• Model: claude-opus-4-8, identical for both conditions. Embeddings: voyage-3. Retrieval: top 5 chunks via POST /search.
• One sample per condition per task; the recorded run is from 2026-07-07.

Scoring

Each task is scored as the fraction of its required API patterns the generated code matches; the headline number is the mean across tasks.

Results

Same model, same tasks — the only variable is whether retrieval put the design system in context.

Limitations (stated up front)

The repo documents these rather than hiding them: N = 12 tasks; a single model; regex-based grading; the retrieval query is the raw task text (no query rewriting); one sample per condition. The claim is directional and honest, not a leaderboard entry.

Reproduce it

make demo-value

Requires real ANTHROPIC_API_KEY and VOYAGE_API_KEY. The full per-case scorecard lives in docs/PROOF-OF-VALUE.md in the repo.

Retrieval evals

Day-to-day retrieval quality is tracked by an eval suite held to the deterministic fake-embedder baseline — free to run, immune to drift, and safe for CI. Improvements land only when the numbers move.

The usage-weighted boost is an example: at weight 0.002 it moved hit@5 from 0.7903 to 0.8065 and MRR from 0.7032 to 0.7040, so it shipped. See Hybrid retrieval.

The eval gate

Generated UI passes a six-check quality gate in CI before merge — including axe-core accessibility checks, an LLM judge, and visual baselines rendered by the render farm. The gate is documented in docs/EVAL-GATE.md in the repo.