Blinded eval

Leading words: organic prompt, blinding, isolate, artifacts not self-report, inconclusive is not a pass.

The parent owns the experiment. Candidates never learn they are in an evaluation. The judge sees sanitized labels and the rubric, never variant names. Grade from files written and files opened, never from a candidate's claims.

Skip this skill for typos. A behavior-changing skill or rule draft is the trigger.

Non-negotiables

Copy the first items of the matched steps below into the todo list verbatim. A step you skip stays with skip: <reason>.

1. Frame the variant and write the rubric (3–6 criteria). Hold the rubric back from candidates.
2. Isolate each candidate in its own working directory. Plant only organic context. Sanitize names — project-shaped, not experiment-shaped.
3. Author one organic user prompt. Same prompt to every candidate.
4. Spawn N candidates as isolated subagents. Inherit the host model unless the host exposes distinct model families; then split families across candidates. Do not require Cursor arena, cloud workers, or a specific slug menu.
5. Spawn one blinded judge. Prefer a different model family when the host exposes one. The judge sees outputs labeled A/B/C plus the rubric.
6. Read every candidate artifact yourself. Compare to the judge. Grade chain-following from what they wrote and opened, not self-report.
7. Verdict: promote, iterate, or abandon. Inconclusive is not a pass.

Blinding word list and sanitization: references/blinding.md. Judge prompt template: references/judge-prompt.md.

Phase A — Frame

State, to the user only:

• Variant under test (what changed, vs what baseline).
• Success predicate (falsifiable).
• Rubric: 3–6 concrete criteria the judge will score. Example scale: pass / partial / fail per criterion.

Do not put the rubric, the words eval, rubric, candidate, baseline, or variant in any directory, filename, or prompt a candidate will see.

Phase B — Isolate

One directory per candidate (git worktree or temp dir). Never commit these trees into the kit repo.

Plant what an organic task would have: a small project skeleton, the skill folder under test (baseline copy or draft copy), maybe a README. No sibling skills the candidate would not naturally open.

Sanitize:

• Directory names: invoice-cli, changelog-bot — not skill-eval-a.
• Skill folder inside the plant uses the skill's real name (authoring-skills-and-rules), not variant-b.

Phase C — Organic prompt

Write what a user would type. State the goal. Do not ask which skills, principles, or files they applied. Do not mention other candidates.

Example shape: "This repo needs a skill so agents stop inventing changelog sections. Write the skill."

Phase D — Candidates

Spawn N parallel Task / equivalent subagents. Minimum useful N is 2 per variant (baseline vs draft) or 2+2. Give each:

• Absolute working directory.
• The organic prompt only.
• Instruction to write files in that directory and stop. No git commit.

Route bulk to subagents. Keep summaries in this thread, not raw payloads.

If the host cannot spawn isolated subagents, run candidates sequentially in separate directories and still blind the judge. Say so in the verdict.

Phase E — Blinded judge

Fill references/judge-prompt.md. Attach each candidate's written files under labels A, B, C. Never include which model or which variant produced them. Shuffle label assignment; record the mapping only in the parent notes.

The judge returns per-label scores against the rubric and a ranking.

Phase F — Parent synthesis

Read every candidate output end to end. If you disagree with the judge, the rubric is ambiguous or a model is biased — fix the rubric and re-run, do not promote on a coin flip.

Reply contract:

• Variant under test
• Rubric
• Per-candidate notes (files written, what they actually did)
• Judge verdict
• Your synthesis
• Recommendation: promote / iterate / abandon

Do not promote on self-report. A candidate that claims it followed the skill but wrote a 600-line SKILL.md with no references/ fails criterion 3.

Dogfood rule

This skill is proven by using it on a real draft. Do not eval blinded-eval with itself in a loop.