Set Up LLM Evals

An eval is a test suite for behaviour you cannot assert with ===. Without one, every prompt edit, model upgrade or retrieval tweak is a guess, and a regression is found by a user. This skill sets up the smallest harness that catches real regressions: a golden set in the repo, checks that run through the same seam as production, a gate in CI and a record over time.

1. Audit current state

Read .claude/stack-profile.md (languages, tests, ci, ai). If absent, detect per stack-profile.md. Change nothing yet:

ls evals evals/datasets prompts config/llm.models.json 2>/dev/null
grep -rln "promptfoo\|inspect_ai\|braintrust\|deepeval\|ragas" . --include=*.json --include=*.yaml --include=*.toml --include=*.ts --include=*.py 2>/dev/null | grep -v node_modules | head
grep -rn "eval" .github/workflows 2>/dev/null | head
ls src/platform/llm 2>/dev/null                        # the seam: evals call the same code path as production
git log --oneline -- prompts config/llm.models.json | head   # prompt and model changes that shipped without an eval

Report: which features call a model, which have a golden set, whether any check runs in CI, prompt or model changes made without a result, and whether failures seen in production became cases.

2. Decide what to do

• No seam → run build-llm-seam first; an eval that bypasses the seam tests something that never runs in production.
• No golden set → create evals/datasets/<task>-gate.jsonl with 20 to 30 cases and the deterministic checks (steps 5–6). Judge later.
• Golden set but no gate → add the runner, thresholds and CI job (step 6).
• A judge exists but was never compared with human labels → calibrate it (step 7) before trusting any number it produces.
• Retrieval features → add the retrieval eval from build-rag-pipeline first.
• All present and Verify passes → "already in place".

3. Detect the track

Signal	Branch
TypeScript service (default)	In-repo harness: eval-code.md, runs with Vitest and tsx, calls the seam
Python service	Inspect AI (MIT): eval-code.md; the harness there is a port-free alternative
Existing promptfooconfig.yaml	Keep it; add the golden set and the gate rules around it. Do not run two harnesses for one task
Team wants a hosted UI, online scoring, shared experiment history	Braintrust or similar; keep the dataset files in the repo as the source of truth

Why the default is a small in-repo harness rather than a tool: eval-patterns.md.

4. Install only what's missing

pnpm add -D vitest tsx                 # TypeScript (zod is already a dependency of the seam)
uv add --dev inspect-ai                # Python track

No other dependency: metrics are a few functions, results are JSONL.

5. Generate the seams

evals/
  datasets/<task>-gate.jsonl     # 20-30 frozen cases: the PR gate (deterministic only)
  datasets/<task>-full.jsonl     # 100+ cases: nightly and on prompt, model or config changes (judged)
  datasets/labels-<task>.jsonl   # human pass/fail labels for judge calibration
  lib/metrics.ts, lib/run.ts     # recall@k, MRR, kappa; runner, checks, gate
  gate.ts                        # CLI: run a dataset, apply thresholds, set the exit code
  thresholds.json                # floor, allowed drop vs baseline, budget per dataset
  history.jsonl                  # one line per recorded run (committed)
  results/                       # per-run JSONL (gitignored, CI artifact)
prompts/eval-judge.md            # the rubric, versioned like any prompt

Code: eval-code.md. Dataset rules: every case has an id, an input, and expectations that a program can check (must_contain, must_not_contain, must_match). Seed it with real inputs (redacted), every production failure (each bug becomes a case), at least one injection attempt and, for RAG, an unanswerable question. Never put secrets or personal data in a dataset file.

6. Wire it up

1. Add tasks to the runner: eval-gate (tsx evals/gate.ts <task>-gate) and eval-full (tsx evals/gate.ts <task>-full --judge).
2. CI: run eval-gate on every PR that touches prompts/, config/llm.models.json, src/platform/llm/ or the feature; run eval-full nightly and on release branches. The job needs the API key as a secret, so for fork PRs (no secrets available) it is skipped and non-blocking; the deterministic checks still run there through the unit tests (pnpm vitest run tests/evals.test.ts, fake adapter, no network). Job structure and SHA-pinned actions: workflow-templates.md.
3. Set EVAL_RECORD=1 on the nightly job so each run appends to evals/history.jsonl; the gate compares the next run with the last recorded pass rate.
4. Add a budget per dataset in thresholds.json. The runner stops with an error when spend reaches it.

7. Verify

pnpm vitest run tests/evals.test.ts                  # runner, checks, budget stop and gate logic: no network
ANTHROPIC_API_KEY=… pnpm tsx evals/gate.ts tickets-gate; echo "exit=$?"   # one paid run

Expected: the unit tests pass. The gate run prints { total, passed, passRate, costUsd, baseline, problems } and exits 0 when passRate is at or above minPassRate and not more than maxDropVsBaseline below the baseline, otherwise 1. Break a prompt on purpose (delete a rule), rerun, and see the exit code turn 1: a gate that cannot fail is decoration.

Calibrate the judge before relying on --judge: label 30 to 50 real outputs pass or fail by hand, run the judge on the same outputs, compute Cohen's kappa (cohensKappa in lib/metrics.ts). Below 0.6 the judge is not ready: tighten the rubric until it is. Repeat when the judge model or the rubric changes.

References

• eval-patterns.md: rules and why. eval-code.md: all code.
• llm-seam-ts.md: the harness calls the seam.
• llm-security.md: injection cases belong in the set (LLM01).
• stack-versions.md: Inspect AI, promptfoo, Vitest lines.