VacationDeals.to — Evergreen Operations Runbook

Run these phases IN ORDER. Obey the guardrails, then report a consolidated summary.

GUARDRAILS (learned the hard way)

• Agent waves ≤ 5 at a time. A 13-agent batch was killed mid-run by a session limit (2026-07-30). Dispatch writer/research/rate swarms in waves of ≤5, insert + commit each wave before the next. Agents must write once and stop (no self-expansion loops — those burned the tokens that tripped the limit).
• Guarded deploys: always pnpm build on the VPS BEFORE pm2 restart. If the build fails, do NOT restart — the site stays up on the old build. Deploy recipe: cd /var/www/vacationdeals && git pull -q origin main && set -a && source .env && set +a && pnpm build && pm2 restart vacationdeals-web --update-env.
• Cost: DataForSEO creds have been 401 (refresh at app.dataforseo.com); treat volume data as optional. The only metered spend is the AI-reviews cron (guarded to 2 runs/day) + any DataForSEO. Do not exceed a $25 all-in run without asking.
• Programmatically re-validate agent output (word floors, FAQ counts, unique slugs) — a writer once shipped 10 under-length posts while claiming it validated them.

PHASE 1 — Full 3-layer backup (crown-jewel = deal_price_history)

ssh root@72.60.126.82 "/usr/local/bin/vacdeals-db-backup.sh >> /var/log/vacdeals-dbbackup.log 2>&1 && ls -t /root/db-backups/*.pgdump | head -1"
# then scp the newest dump to backups/db/ (local Dropbox mirror — it drifts stale)
# then: bash scripts/backup-to-private-github.sh   (private GitHub repo)

Verify deal_price_history row count + span before/after (it should only grow).

PHASE 2 — Recrawl all sources + maintenance

ssh root@72.60.126.82 "nohup /var/www/vacationdeals/scripts/run-with-env.sh /usr/bin/npx tsx src/scrape-wave.ts --wave=all > /var/log/vacdeals-fullscrape-$(date +%Y%m%d).log 2>&1 &"
# wait for all 5 waves green, then run maintenance (paths are relative to apps/scraper):
#   run-with-env.sh npx tsx src/check-deal-health.ts       (zombie sweep + dead-URL)
#   run-with-env.sh npx tsx ../../scripts/nightly-data-quality.ts --fix
#   run-with-env.sh npx tsx src/verify-prices.ts --limit=50

PHASE 3 — Rate spot-check swarm (≤5 agents)

Dump active deals to a scratch PSV, then dispatch ≤5 agents grouped by source. Each samples deals across its sources, WebFetches the partner lander (curl fallback for JS/403), and returns a table: deal_id|scraper_key|db_price|live_price|verdict|notes (MATCH/MISMATCH/GONE/FETCH_BLOCKED). Fix confirmed data errors in the DB (stale prices, credit-as-price, expired/repurposed URLs, placeholder original_price=50000). Target ≥20 rates checked. Confirms scrapers work AND published price == brand-site price.

Known-walled sources (curl/WebFetch always FETCH_BLOCKED): vegas-timeshare (sgcaptcha), holiday-inn (403 WAF), hyatt (JS-rendered price), branson-travel-group (Cloudflare/sgcaptcha). Their CRON scrapers DO work (Playwright/desktop-UA paths) — as of 2026-09-23 all four had fresh active deals — so FETCH_BLOCKED here is a verification gap, NOT a data problem. Don't deactivate them on FETCH_BLOCKED alone. To actually verify their prices, give the agent handling that chunk the stealth-browser-mcp tools (spawn_browser → navigate → get_page_content), which pass these challenges; fall back to reading the DB scraped_at freshness as a liveness proxy when the browser isn't available.

PHASE 4 — Deep-research swarm: new vacpack sites (≤2 agents)

Dispatch research agents to find NEW timeshare-preview sites we don't scrape (exclude the ~48 current sources — see CLAUDE.md/scraper list). Each returns a ranked list: url | example packages+prices | presentation required? | Cheerio vs browser | bot-blocking. Relay the ranked candidates; building crawlers is a follow-up (don't auto-build).

PHASE 5 — 50 bespoke Google-Discover blog posts (≤5 agents, last wave)

Follow the STYLE-SPEC in research/blog-batches/weird-batch-2026-07/STYLE-SPEC.md (BLUF box, 8-10 AEO FAQs, humanization in BODY ONLY, casual first-person, real vacpack data). NEW angles each time — check existing slugs first (SELECT slug FROM blog_posts) to avoid dupes. Discover-oriented = curiosity-gap titles, timely/story hooks, strong first image; lightly SEO'd (one keyword woven in). Author as JSON to a new research/blog-batches/<batch>/ dir, validate (unique slugs, ≥900 on-page words incl. FAQs, ≥6 FAQs), commit + push, insert on the VPS, rebuild (guarded) so the sitemap picks them up, verify a sample renders 200 + FAQPage schema.

Insert command gotcha: insert-blog-batch-json.ts imports the @vacationdeals/db workspace pkg. npx tsx AND pnpm exec tsx from the repo root BOTH fail (MODULE_NOT_FOUND / "tsx not found"). Run it from apps/scraper (which has both tsx and the db symlink) with ABSOLUTE paths: cd /var/www/vacationdeals/apps/scraper && set -a && source /var/www/vacationdeals/.env && set +a && ./node_modules/.bin/tsx /var/www/vacationdeals/scripts/insert-blog-batch-json.ts /var/www/vacationdeals/research/blog-batches/<batch> [--dry-run]. Always --dry-run first (validates required fields, ≥5 FAQs, category enum, skips existing slugs), then run for real.

Sitemap-cache trap (learned 2026-08-11): sitemap.ts has revalidate=3600, so new posts appear in /sitemap.xml within the hour on their own — DO NOT try to force it by deleting .next/server/app/sitemap.xml*; that removes the compiled route and makes the sitemap 500 (MODULE_NOT_FOUND), needing a full TURBO_FORCE=true pnpm build to recover. If you must refresh it now, a normal guarded rebuild + pm2 restart regenerates it cleanly. The per-slug blog pages go live immediately (fresh ISR), only the sitemap lags.

FINISH

• Update docs/NEXT-ENHANCEMENTS.md (dated entry) + relevant memory.
• Report a consolidated summary: backup status, recrawl/maintenance results, rate spot-check verdicts + fixes, new-site candidates, blogs published + live count.