score-loop

You run one metric up (or down) unattended, one hypothesis per commit, and log every experiment to skilleddocs/experiments.md. Only for a score you can rerun and that is a number; an unrepeatable or subjective score → say so and stop.

Commands: setup (default; also when experiments.md has no setup header), resume, status, summary.

1. Setup (confirmed once)

1. Fill the header below from the request and the repo; ask only for what's missing. Create the loop branch off the current HEAD.
2. Harness first. No local harness that mirrors the grader → write one and commit it before experiment 1. It runs a correctness check (pass/fail) and the score N times, and prints the median. It must defeat caching/memoisation: fresh inputs or a fresh process per timed run, and the correctness check never warms the timed call. A near-zero or implausible score is invalid unless correctness proves otherwise.
3. Noise margin: run the harness once on the baseline; propose margin = (max − min of its N runs) / baseline median, as a %.
4. Calibrate (after the gate): one official submission of the baseline; record the local–official gap. A gap wider than the margin (as a % of the baseline) → fix the harness before experiment 1. No official grader → skip and write calibration: none (no official grader).
5. Freeze time = deadline − lead, lead = max(10% of remaining, min(30 min, 50% of remaining)) (≥5h left: 10%; 1-5h: 30 min; <1h: half the remaining time); no deadline → no freeze. One experiment (one harness run, as timed on the baseline) plus the calibration must fit before freeze; otherwise refuse, say so, and stop.
6. Gate: show the filled header, how the score could be gamed and how the harness blocks it (proxy-metric gaming: the score improves without the real goal improving, e.g. work moved outside the measured region or special-cased fixed inputs; the correctness check, fresh or hold-out inputs and timing the whole call guard it), and each outward action (the calibration submission, the quota, any paid step). One explicit yes covers this setup; anything vaguer is a no. Then write the header, run the calibration, and go unattended.

# Score loop: <slug>
- metric: <name>, <higher|lower> is better
- grader: <official grader cmd/URL, or none>
- harness: <local cmd>  (correctness check + median of N=<n>)
- submission: <path or cmd>
- quota: <official submissions pre-approved, or 0 = ask each time>
- spend cap: <amount, or 0 = no paid steps>
- deadline: <local datetime, or none>  freeze: <datetime>
- noise margin: <p>% of best  (spread <s> / baseline median <m>, from <n> runs: <values>)
- calibration: local <x> vs official <y>
- K: 5   parallel: 1
- branch: <prefix>score-loop/<slug>   resume: /score-loop resume

## Ledger
| n | time | commit | hypothesis | local median | official | keep/revert | note |
|---|---|---|---|---|---|---|---|

## Next ideas
- <idea>

## Tried
- <idea>: <why it failed>

<prefix> is the user's branch prefix with its slash (from their CLAUDE.md), or empty. Commit experiments.md or leave it untracked per TRACKING.md.

2. Loop

Per experiment:

1. Pick the most promising idea from Next ideas; the list empty → add 3 from what the ledger shows so far and Tried (not repeating Tried ones).
2. Make one change for that one hypothesis; commit exp <n>: <hypothesis>.
3. Run the harness (local runs are ungated).
4. Decide, then append one ledger row:
   • Correctness check fails → git revert --no-edit <sha>, invalid; not counted toward K.
   • Gain = the median's relative improvement over the best median (direction-aware); m = the margin.
   • Gain > m → keep; it is the new best.
   • m/2 < gain ≤ m → borderline: rerun both now, interleaved so drift hits both alike: N single runs each in ABBA order (best, candidate, candidate, best, repeated), which cancels steady drift exactly (harness at N=1; correctness already passed), and take each fresh median. Candidate's fresh median beats the best's fresh median by more than m/2 → keep; else git revert --no-edit <sha>, noise. The fresh medians only decide this row; the best stays unless the candidate is kept (then its fresh median is the new best). Note: borderline: fresh best <b> (was <best>), fresh cand <c> (was <cand>), gain <g>% vs m/2 <h>%.
   • 0 ≤ gain ≤ m/2 → git revert --no-edit <sha>, noise.
   • Worse (gain < 0) → git revert --no-edit <sha>, revert.
5. A failed idea moves from Next ideas to Tried with one line why. The user may append ideas at any time; read the list fresh each experiment.

Ledger rows are append-only: reverted rows stay, a correction is a new row.

Parallel (parallel: N > 1, opt-in): spawn N subagents, each with its own worktree and branch off the loop branch and one idea. Each runs the harness and returns its median and correctness. One row per variant; merge only the best qualifying variant into the loop branch, then remove the worktrees. Graders that share the machine get noisier: say so at setup.

3. Outward actions

• Official submission: only inside the setup quota; past it, show the exact submission and wait for an explicit yes. Record the result in the row's official column.
• Paid (submission, compute, API): through spend-gate, which takes its own yes per action; record the cost in the row's note. The spend cap is a ceiling, not approval.
• Push or merge: only on an explicit yes to that push or merge.

4. Stops

Check before each experiment:

• Budget: the next experiment needs a paid step that would take total spend (sum of costs in the ledger) past the spend cap. Cap 0 → any paid step stops the loop; unpaid experiments never hit this stop.
• Deadline: freeze time reached → no new experiments; rerun the harness on the best commit, show the final submission and submit on a yes, then §5.
• Plateau: K consecutive experiments without a keep (invalid rows don't count).

On any stop: §5, then report why it stopped.

Context degrading (smartzone warning, or the ledger no longer fits your head): finish or revert the current experiment, then call handoff-auto with "continue the score loop from skilleddocs/experiments.md: /score-loop resume", and stop.

5. Summary

From the ledger only, no reruns: kept commits ranked by relative gain (gain / best before them), not absolute, reverted ideas with why, noise rows, invalid rows, best local and official score. Append it to experiments.md under ## Summary <date> and print it. Ablation reruns only when the user asks.

Commands

• resume: read the header and ledger, switch to the loop branch. HEAD is an exp <n>: commit with no ledger row for n → evaluate it as that experiment. Recheck the stops, then §2. No new setup gate.
• status (read-only): best (commit, local, official), experiment count (kept / noise / revert / invalid), time to freeze and deadline, spend and quota left, next idea.
• summary: §5 now.

Backlog of issues instead of one metric → that is meta-orchestrator, not this skill; never invoke it from here. Live session status → sitrep.