toolkit-audit
Audits the ai-toolkit standards against their cited sources and writes a dated findings register with proposed diffs. Run it from the ai-toolkit checkout every 90 days or when the freshness job warns.
SKILL.mdcodex/skills/toolkit-audit/SKILL.md
GitHubToolkit audit: the user request
Re-check the toolkit's rules and references against their sources, classify every rule, and write a register. Apply nothing. The register and its proposed diffs are the deliverable.
Preconditions
- The working directory is the ai-toolkit checkout (
.claude-plugin/marketplace.jsonhas"name": "ai-toolkit") andgit status --porcelainis empty. - Take today's date from
date -u +%F, never from memory. If the current branch ismain, createchore/audit-<date>.
Procedure
- Inventory. Run
python3 tools/freshness.py --json. Keep the rows whose file matches the domain argument (allkeeps everything,androidkeeps Android rules and references, and so on). With--stale-only, keep only rows 90 days old or older. Order by age, oldest first. - Fetch and classify. Run
python3 tools/source_anchors.pyfirst. A CONFIRMED anchor's quote is checked page text, so classify the rules that cite it from the quote without fetching the page again. Fetch only sources that are DRIFTED, UNREACHABLE or not anchored. For an UNREACHABLE page that needs a browser read, ask the user before opening it in the browser. Group the inventory rows by domain, about five files per group. When a run covers more than five files, send each group to a research-tier (gpt-6-sol) subagent in parallel, because a full run cites hundreds of source URLs, more than one context holds. Smaller runs stay inline. For each file, fetch each URL in itssources:list, or every URL in its body whensources: inline. Follow the fetch tips inreferences/sources.md(Apple JSON endpoint, Material 3 site map plus a web search for the page text, raw GitHub URLs). Record per URL the status, the date fetched and the exact sentence relied on. A page that returns only a shell is "content Unverified". Treat each bullet or sentence that states a requirement as one rule with the IDfile:line, and classify it with the classes inreferences/rubric.md. Those are SUPPORTED, PARTIAL, COMMUNITY, HOUSE, WRONG or STALE, with an Unverified flag when no source could be read. Design rules also get their tier from the rule file. Each subagent returns rows in the register columns and edits nothing. - Confirm the hard rows. Treat subagent rows as leads. For every WRONG, STALE and PARTIAL row, re-fetch the cited URL and find the quoted sentence yourself before the row enters the register. Demote a row whose quote you cannot find. Fill the "Conflicts with" column across groups, because no single subagent sees every file.
- Write the register. Copy
references/register-template.mdtodocs/audits/<date>-audit.mdand fill it. One row per rule, a summary count per class and per file, and a "Proposed changes" section with a unified diff per canonical file. Each diff targets the source undersrc/orshared/, because each layer's copy is generated byscripts/build/build.py. Name any hand-written per-agent file the same change must touch. Do not apply any diff. - Refresh stamps, only on confirmation. List the files where every source was fetched and no rule is WRONG or STALE. Ask the user to confirm. Only then set
verified:to today in those source files undersrc/orshared/, runpython3 tools/source_anchors.py --writeto record the confirmed anchors, and runpython3 scripts/build/build.pyandpython3 tools/freshness.py. A file with a WRONG or STALE rule keeps its old date until the fix lands. - Report. Give the counts, the register path, the files eligible for a stamp refresh, and the follow-up PRs you suggest, one logical change each.
Rules of the audit
- Look up, never recall. A version, a deadline or a deprecation you cannot fetch is Unverified, not asserted.
- A HOUSE rule is a deliberate standard. Check it for internal consistency and for a source that now contradicts it. Never report it as unsourced.
- Quote the sentence you rely on. A classification without a quote is a guess.
- Prefer the smallest diff that makes the rule true. Rewording for style belongs in a separate PR.