skill-health — Skill-Library Structural Debt Scan

Detect skill technical debt — library-level defects that do not break a single skill in isolation but degrade the library over time (SkillOps, Pu/Song/ Zhao 2026, arXiv:2605.13716). This skill owns the one debt pattern no other harness skill checks: missing artifacts — a SKILL.md that references a scripts/ module, a bash script, an agent, or a sibling skill that no longer exists on disk (it silently dangles after a rename or delete).

This is a structural property — decidable from the literal text plus a filesystem exists() check — so deterministic code owns it. The semantic, risk, and validation dimensions are delegated, never re-implemented here.

Only dangling is a defect claim

The scanner (--help lists flags, --json the keys) reports three lists:

• dangling (exit code 1) — a written path that does not exist. This is the only defect claim.
• unresolved_names — skill / command names written without a path that match no file under the skills root. An enumeration handed to judgment, not a defect: the name may be a CLI builtin, a plugin command, or a skill in another repo, and the slash-command namespace is not enumerable from disk. _KNOWN_NON_FILE_SKILLS in the scanner is noise reduction, not an authority.
• external — skills whose directory is a symlink out of the skills root. Also an enumeration, not a defect: it answers "would a fix applied here survive?" — it would not, since the owning tree overwrites it on upgrade. Route these upstream (issue / PR). References inside them are still scanned.

Boundary (read first — this skill does not overlap its neighbours)

SkillOps frames library health as four dimensions. This harness already covers three; skill-health adds the missing structural one and federates the rest:

Dimension (SkillOps)	Owner	skill-health does
Compatibility (refs resolve)	skill-health ← here	the deterministic scan (Phase 2)
Utility (frequency, value)	skill-stocktake (holistic Keep/Improve/Retire/Merge)	read its signal; flag over-specialized (LLM)
Risk (security, side effects)	/claude-security plugin	delegate; prompt to run if stale
Validation (tests/consistency)	skill-comply / skill-creator	note which skills lack a validator

Phase 1 — Inventory

Enumerate skill definitions with Glob (~/.claude/skills/*/SKILL.md; plus {cwd}/.claude/skills/*/SKILL.md if the project has local skills). State up front how many skills will be scanned.

Phase 2 — Compatibility scan (the deterministic core)

Run the structural reference scanner. It extracts every explicit local reference from each SKILL.md (python -m scripts.X, bash …/x.sh, ~/.claude/agents/X.md, and Markdown links to local files / sibling skills) and reports those whose target does not exist:

Paths below start at ${CLAUDE_SKILL_DIR}, the directory holding this SKILL.md; an agent that does not substitute the variable reads it as that directory.

uv run --frozen --project "${CLAUDE_SKILL_DIR}" \
  python -m scripts.scan_refs ~/.claude/skills --json

--external-urls on the same scanner lists the external URLs the corpus names, fence- and placeholder-aware, and exits 0 — it is the input to this skill's other probe, python -m scripts.url_liveness --urls-from - (URL reachability as live / dead / blocked / skip; blocked is not dead). Reachability is checked by whoever owns the audit, once and serially, never inside parallel batch agents — skill-stocktake Phase 1 is the caller.

Exit code is the code-owned gate: 0 clean, 1 dangling references found, 2 scan root missing. The scanner is conservative by design — it skips template placeholders (<your-repo>/x.sh), illustrative example links ([](url)), and --directory-overridden commands, because a false "missing artifact" is worse than a missed one. Run --help for flags; omit --json for a human report.

The same run also reports external skills (external in JSON). Surface these before any repair is proposed — they set where a fix can go — and name the owner of each (handling per external above).

Present each dangling reference with: skill, ref type, the raw reference, the resolved path, and the line. Do not auto-fix — a dangling reference may mean the artifact was deleted (remove the reference) or renamed (repoint it) or, for a ../-escaping link, that the skill was authored in a repo and vendored into the harness (a portability issue per the skill-creator skill's references/portability.md). The repair is a human judgment; surface the fact, let the user decide.

Phase 3 — Federate the other three dimensions (read, don't re-implement)

For the scanned skills, surface the existing signals so the report is a single health view — labelling each value's source:

• Utility — do not read ~/.claude/metrics/skill-usage.jsonl by hand. The corrections that decide whether its numbers mean anything live in usage_stats (in the sibling skill-stocktake skill, when it is installed next to this one); run the script once per window:

  uv run --frozen --project "${CLAUDE_SKILL_DIR}/../skill-stocktake" python -m scripts.usage_stats --days 90
  

  Where a skill is rarely or never triggered, judge [LLM] whether the cause is over-specialized scope (a trigger so narrow it never fires) versus simply new — last_used is not clipped to the window, so "used 100 days ago" is distinguishable from "never". If measurable is false, or span_shorter_than_window is true, render usage as unmeasured — never 0.

  Enumerate residency-fold candidates: skills with nonzero context (the /skill-doctor residency column; - means the description is already off the listing and there is nothing left to fold) and zero deliberate use in the window. Their description resides in every system prompt without having driven a single selection — the description is the fold candidate. Hand the list to the author with the decision axis attached (default: add an explicit one-line reference from a related skill / rule, then disable-model-invocation: true — even a one-line description still resides in the system prompt as an unaudited instruction; a one-line description is the fallback only when no reference site exists); exclude skills whose window is shorter than their age would need (span_shorter_than_window, or added days ago). This is an enumeration, not a verdict — folding is the author's call.

• Size — enumerate SKILL.md bodies over the 500-line limit (wc -l ~/.claude/skills/*/SKILL.md | sort -rn; limit per skill-creator §3, sourced from Anthropic's official best practices as of 2026-08-29 — re-check on doc updates). Growth after creation is the case the creation-time gate cannot catch. The verdict on whether an oversized body is bloat or load-bearing belongs to skill-stocktake's Hygiene question, not here.

• Risk — point to the latest CLAUDE-SECURITY-*/CLAUDE-SECURITY-RESULTS.md if present; if none/stale, recommend running /claude-security. Do not re-scan for vulnerabilities here.

• Validation — note (missing validators debt) which scanned skills have no skill-comply spec and no record of passing skill-creator's gates — the fresh-context draft verdict (its §4) and, for a skill with verifiable output, the native ablation screen claude plugin eval <skill dir> --ablation with-without --runs 3 (its §5) — i.e. no way to verify their behaviour. Judge [LLM] trigger↔body consistency only where it is in doubt.

Not measured: skill success rate (did using the skill improve the outcome, not just fire?). It needs counterfactuals the harness does not capture; left as a known gap rather than a fabricated number.

Phase 4 — Report & ledger

Render a skill × dimension table: Skill | Compatibility | Utility | Risk | Validation, where Compatibility shows the dangling-reference count (the only hard number), the others show the federated signal or unmeasured. List each dangling reference with its repair options (remove / repoint / portability).

Persist the scan to ${CLAUDE_SKILL_DIR}/results.json (the --json output) with a real UTC scanned_at (date -u +%Y-%m-%dT%H:%M:%SZ), so the next run can diff. Update it inline with Read/Write.

Related

• Your publish step (the author's harness uses harness-sync) — use it to publish this skill to a public repo.