Skill: reanalyze

Re-run analyze-run.sh on every run matching an RQ selector, reaggregate via aggregate-by-query.py, and propose findings updates. No new runs are started — this skill only refreshes existing data.

Typical use cases:

• Pipeline fix (analyze-run.sh changed, verification adapter added)
• Manual run cleanup (crash runs deleted, runs re-added)
• ESLint/SonarJS config change affecting smell metrics
• New fields added to metrics.json

Argument

• RQ-N (e.g. RQ-prompt-known-kata) or a direct path to an RQ dir.
• If not given: ask back ("Which RQ? e.g. RQ-prompt-known-kata").
• RQ dirs live in four subtrees: research/questions-claude/<chapter>-*/, research/questions-opencode/<chapter>-*/, research/questions-cross/<chapter>-*/, and research/workflow-dev/<chapter>-*/. The chapter prefix is an ordering label; the stable id is the frontmatter id: (a slug like RQ-prompt-known-kata). Resolve it to a path by exact-line id-grep across all subtrees:

  RQ_DIR=$(grep -rlE "^id:[[:space:]]*RQ-prompt-known-kata[[:space:]]*$" \
             research/questions-claude/*/README.md \
             research/questions-opencode/*/README.md \
             research/questions-cross/*/README.md \
             research/workflow-dev/*/README.md \
           2>/dev/null | head -1 | xargs -r dirname)
  

Phases

Run sequentially. On errors in any phase, stop and ask the user, do not skip ahead.

---

Phase 1 — Resolve & Match

1. Resolve the RQ path into $RQ_DIR via the id-grep in "Argument" above. On no match, ask the user; on multiple, take the first and inform.
2. Read $RQ_DIR/README.md — parse frontmatter for controls and factors (needed to build the selector).
3. Find all matching runs in experiments/runs/:

   for d in experiments/runs/*/; do
       jq -e 'select(
           .kata == "<expected>" and
           .workflow == "<expected>" and
           (.model | test("<model-pattern>"))
       )' "$d/metrics.json" > /dev/null 2>&1 && echo "$d"
   done
   

   The selector logic mirrors aggregate-by-query.py: kata = kata_base + "-" + prompt for each prompt value (from factors or controls), model from factors or controls, workflow from controls. Match against .kata, .workflow, .model in each run's metrics.json.
4. Report: "Found N runs matching RQ-X selector."
5. If N == 0: STOP — "No matching runs found. Check that experiments/runs/ contains runs with the expected kata/workflow/model."

---

Phase 2 — Reanalyse

1. Get user confirmation: "Reanalyze N runs? This re-runs analyze-run.sh on each (ESLint, verification, metrics extraction). Estimated time: ~N × 10s."

2. After "yes": reanalyze inside the container, never on the host:

   ./experiments/reanalyze-in-container.sh --rq "$RQ_DIR"      # every run in runs.csv
   ./experiments/reanalyze-in-container.sh <run_dir> [<run_dir> ...]   # a subset
   

   The wrapper runs analyze-run.sh in docker-batch:latest with the same mounts the batch uses, and pays container startup once for the whole set. It prints one line per run and a final reanalyzed: N failed: M; a single failing run does not abort the rest.

   Do not call ./experiments/analyze-run.sh directly on the host. The verification stage shells out to the package manager, so a host pnpm that differs from the Dockerfile pin (pnpm@9.15.9) makes the CLI never start: every scenario fails, verification_pct is rewritten to 0, and the host pnpm mutates the run's node_modules besides. The number that lands is plausible enough to survive review and reads as a correctness collapse. analyze-run.sh now refuses to write in that situation (it exits 3 and leaves metrics.json untouched), but the guard keys on ERR_PNPM_ specifically — other environment drift (node major, ESLint resolution) would still pass through silently. The container is the actual protection.

   Reanalysis is compared against runs that were not reanalyzed, so the analysis environment has to match the batch environment exactly. That is the whole point.

3. Report: "Reanalyzed N runs."

4. Recompute cost_usd — mandatory after any reanalysis of pi runs, not optional. analyze-run.sh reads cost_usd from transcript-metrics.json and writes it whenever it is not null. For pi/Requesty runs that source holds 0 (pi's cost scaffold; Requesty reports no inline cost), so the step above overwrites correct costs with zero — silently, with no error. Restore them:

   ./experiments/compute-cost.py "$RQ_DIR"
   

   Then verify: no cell may show cost_usd 0.00 at non-zero total_tokens. That combination is impossible and means the step did not take.

5. If mutation_score is in the RQ's outcomes:, also run ./experiments/compute-mutation-score.py "$RQ_DIR" (expensive — minutes per run; only for tests_passing = true).

---

Phase 3 — Reaggregate

1. Run:

   ./experiments/aggregate-by-query.py "$RQ_DIR"
   

2. Read the generated summary.md.
3. Show the cell coverage table to the user.
4. Show the primary outcome pivot table (typically verification_pct or the first outcome in the frontmatter).

---

Phase 4 — Findings update (write-first)

Write directly to findings.md, then notify the user to review. Markdown tables and trophy assignments are much easier to evaluate as rendered output than as a chat proposal; reverting is cheap (it's only markdown). After writing, send one line (in the user's language), e.g. "written — please review", with a brief list of what changed (STALE / STATUS / NEW).

Exception: deletions of existing findings still require explicit user confirmation before the Edit — losing a documented finding is more expensive than re-reading a fresh write.

findings.md shows only the current state. No legacy comparisons, no "previously X" references, no "revised"/"confirmed" tags. Header form: ## F-x.y — Title (no trailing suffix). The namespace before the last dot may carry dots itself (F-4.4.1, F-1.12.5 are valid). Keep the em-dash —: the snapshot generator parses on it and drops non-matching headers silently.

Trophy convention (🏆) in overview tables: When refreshing the ## Overview table at the top of findings.md, append 🏆 to the best value per outcome row alongside the bolded winner. Metric direction matters — note it in the column header or row label (smell_total, complexity metrics → "lower = better"; refactorings_applied, predictions_correct_rate → "higher = better"). Award 🏆 only where the spread is meaningful — if all values lie within 1 σ and the framing is "no effect", either award 🏆 to all tied or to none, don't fabricate winners from rounding noise. Trophies are for findings docs only; workflow files stay emoji-free.

1. Read current findings.md and fresh summary.md.

2. For each existing finding (## F-x.y):

   • Extract the numbers referenced in the finding (mean, σ, n, rates, spreads).
   • Look up the corresponding cell(s) in the fresh summary.md pivot tables.
   • Classify:
     • OK — numbers match (within rounding tolerance of 0.01).
     • STALE — numbers have changed. Show: F-x.y: <metric> was <old>, now <new>.
     • STATUS CHANGE — the status tag should change based on new data (e.g. n grew from 3 to 5 → "⚠️ bedingt" can become "✅ stabil"; or a previously stable pattern broke).

3. Check for NEW patterns not yet captured in findings:

   • Scan the pivot tables for cells with notable deviations (σ > 0.3, mean < 0.5 on verification_pct, or cross-cell spreads > 20 pp).
   • If a pattern is new and not covered by any existing finding, propose a new F-x.y block.

4. Apply the changes directly:

   • STALE: replace the affected numbers in the existing finding block via Edit. Rewrite the Datenbasis table and any prose that references the changed numbers. Do NOT add "previously X" or "corrected from" — just write the current state.
   • STATUS CHANGE: update the status tag inline.
   • NEW: append at end of findings.md, after the last --- separator.
   • Update the overview table at the top of findings.md to reflect any changed numbers.
   • Deletion (data contradicts a finding): ask the user first, then remove the block including its --- separator on confirmation.

5. Verify every number you just wrote before reporting. Do not skip this — a wrong number in a findings table is the most expensive kind of error here, because it looks authoritative and gets quoted onward. Pull the written rows back out and diff them against summary.md:

   grep -n "<cell-name>" "$RQ_DIR/findings.md" | grep "|"
   

   Check each value against the same metric in summary.md for this RQ.

   The failure mode this catches: when several RQs are edited in sequence, values from a different kata sit in context and get written from memory instead of looked up. Cell names and metric names are identical across RQs — only the ranges differ, and a foreign value is often plausible enough to survive proofreading. (Real case, 2026-08-04: longest_function = 15.0 from the game-of-life RQ landed in the claim-office RQ, where 21.4 was correct — wrong trophy, plus an interpretation sentence built on the wrong figure.)

   Two habits prevent it upstream: query metrics completely per RQ rather than selectively (the gap appears when a table column is missing from the fresh extract), and treat magnitude as a sanity anchor — claim-office is the large kata (code_mass 666, cycle_count ~47), game-of-life the small one (144, ~15).

6. After verifying, send one short line summarizing what changed:

   written — STALE: F-x.y, F-a.b · NEW: F-x.z · OK: F-c.d, F-e.f. Please review.
   

Discipline-metric comparability: if the RQ declares tdd_discipline or any of its Skip-fed siblings (tdd_discipline_test_first, tdd_discipline_closure, test_first_rate, skip_events, cycles_total, chain_deviations), check whether its cells span the test-list boundary — aggregate-by-query.py warns and names both groups. Those columns then measure the architecture rather than the discipline, so they are reported within a group and never carry a trophy across one. Full argument: README "Comparability: the test-list boundary".

Glossary discipline: use terms from the README glossary ("Code Mass (APP)", "Production LoC", "Correctness (external)", etc.) or metric IDs in backticks. Synonyms like "Code-Volumen" or "LoC-Größe" are forbidden.

---

Abgrenzung zu /run-rq

Aspect	/run-rq	/reanalyze
Starts new runs	Yes (Docker batch)	No
Calls analyze-run.sh	Only on new runs (inside container)	On ALL matching runs (inside container, via reanalyze-in-container.sh)
Aggregates	Yes	Yes
Proposes findings	Yes (Phase 6)	Yes (Phase 4)
Use case	Fill missing replicates	Refresh metrics after pipeline fix

Out of scope

• Starting new experiment runs (use /run-rq for that).
• Cross-RQ aggregation or overview snapshots (use /build-overview).
• Editing prompts, workflows, or verification suites.
• Auto-commit/push — stays a user decision.

Behavior on errors

• Phase 1 no matches: suggest checking the frontmatter selector against actual run metadata.
• Phase 2 analyze-run.sh fails on a run: report the specific run dir and error, continue with remaining runs (do not abort the whole batch).
• Phase 3 aggregate fails: show output, do not proceed to Phase 4.