Site Crawler Stinger

You are equipping site-crawler-wasp-drone, part of the Website Auditor by Legion Code Inc. plugin. Full scope and acceptance criteria: prd-007-site-crawler.

Purpose

Runs a platform-aware crawl of the audited site, up to 100 pages, storing raw HTML and a Markdown extraction per page under site-data/<slug>.html / <slug>.md, plus a site-data/manifest.json index. This Stinger's output is the shared, write-once data layer that every one of the nine Wave-5 audit Drones (technical-seo, aeo-audit, content-semantics, internal-linking, visual-funnel, accessibility-audit, web-security-posture, analytics-stack, performance-cwv) reads read-only, with zero write contention between them, per PRD-007's shared workspace contract.

When to use

• Wave W4, sync, immediately after stack-fingerprint-wasp-drone has written _shared/target-profile.json (this Drone's stack-type dependency, per the build plan's dependency graph and PRD-007's Depends on: prd-003)
• Any re-crawl after a mid-engagement site restructure is suspected (full re-crawl, not a patch, see guides/05-politeness-and-scope-limits.md)
• Never invoke this Stinger for a single-page check; that is stack-fingerprint-stinger's scope

When not to use

• Not for classifying the site's technology stack. That is stack-fingerprint-stinger, which runs first and writes the target-profile.json this Stinger reads.
• Not for analyzing what was crawled (SEO, AEO, accessibility, security, performance, content, linking, funnel, or analytics findings). This Stinger only fetches and stores; every Wave-5 Drone owns its own analysis of site-data/.
• Not for crawling authenticated or gated areas, submitting forms, or exceeding 100 pages without an explicit user-approved re-run. See PRD-007's Non-Goals.

Procedure

1. Confirm _shared/target-profile.json exists with non-null platform, rendering, and confidence. If missing, stop and hand back to the orchestrating agent.
2. Read the platform value and select the seed-path strategy for it (guides/02-platform-traversal-strategies.md).
3. Run shared/scripts/crawl-extract.py (or its algorithm) to crawl up to 100 same-domain pages, breadth-first, respecting robots.txt and rate limits (guides/01-crawl-procedure.md, guides/05-politeness-and-scope-limits.md).
4. Store each page as site-data/<slug>.html + site-data/<slug>.md using the deterministic slugify algorithm, and write site-data/manifest.json as the single index (guides/04-storage-and-manifest-convention.md).
5. Record every unreachable URL in the manifest's unreachable[] array with a reason; never retry into a block (guides/03-fetching-and-rendering.md).
6. Write a short run summary into this engagement's _shared/run-ledger.json entry for this Drone (pages fetched, pages unreachable, platform strategy used). Do not write into this plugin repository's own library/.

References map

• references/templates/manifest-schema.md, load when building or consuming manifest.json and the exact field contract is needed.
• references/templates/site-data-manifest.example.json, load as a worked example of the manifest shape before writing or parsing one.
• references/research/distilled-site-crawler.md, load when a fetching/rendering/platform- detection claim needs verification or a dispute needs settling.
• Source research archive (not shipped with the plugin): the shipped distillation summarizes it; do not require it.
• references/scripts/README.md and shared/scripts/crawl-extract.py, load/run to actually execute the crawl.
• guides/01-crawl-procedure.md through guides/05-politeness-and-scope-limits.md, load per-verb as described in each guide's own heading.

Related drones and stingers

• stack-fingerprint-stinger - runs before this Stinger in wave W1a, writes _shared/target-profile.json, this Stinger's binding platform-classification input.
• technical-seo-stinger - one of nine Wave-5 Drones reading site-data/ read-only; consult when a manifest or storage-convention question is actually a technical-SEO scope question instead.
• aeo-audit-stinger - another Wave-5 reader of site-data/.
• content-semantics-stinger - another Wave-5 reader of site-data/, the primary consumer of the .md extraction quality caveats in guides/03-fetching-and-rendering.md.
• internal-linking-stinger - another Wave-5 reader of site-data/, and the most likely consumer of manifest.json's unreachable[] array as evidence.
• visual-funnel-stinger, accessibility-audit-stinger, web-security-posture-stinger, analytics-stack-stinger, performance-cwv-stinger - the remaining four of the nine Wave-5 Drones reading site-data/ read-only.
• keyword-intelligence-stinger - runs one wave earlier (W3, not W4) and does NOT read site-data/, since it has not been written yet at that point in the sequence; see that Stinger's guides/03-tier-3-ai-inference.md for how it independently sources content instead.

Critical Directive

• You must read all files and context contained within your skill.
• In the event your core knowledge does not provide sufficient guidance you must make every attempt to search the internet, related knowledge base documentation files, and other available resources to supplement your knowledge prior to proceeding with your task.
• Additional related skills can be found here:
  • site-crawler-wasp-drone - this Stinger's paired Drone.
  • stack-fingerprint-stinger - upstream dependency, writes target-profile.json.

Ship Gate

Ship Gate not applicable to this Stinger's own runtime procedure. This Stinger's output (site-data/) is written into the external customer's audit workspace (www.<domain>-audit/site-data/), not into this plugin repository's own tracked source, so a site-crawler run produces no committable code inside this repo for security-stinger, quality-stinger, or github-repo-health-stinger to gate. The Ship Gate does apply, unmodified, to any change made to this plugin repository's own files, including this SKILL.md, its guides, its references, and shared/scripts/crawl-extract.py, before that change is committed and pushed.