Bright Data - Scrape

Run the gates below to pick the product yourself - never ask the user which. State the pick in one line, and one word from the user overrides it.

The gates - run in order, first yes wins

Check	If yes
The marketplace check - already collected. All three at once: a huge generic corpus, needed once, and a few months old is fine. Anything less falls through to gate 1.	The datasets skill. Buying the download is not scraping, so it has its own skill.
Gate 1 - library. A ready scraper already returns these fields. Scrapers come first. Popular platforms resolve from the bundled top-25 table with zero API calls, and only an unlisted site goes to the live catalogue. The input is not always a URL - variants take usernames, hashtags, keywords, locations and more.	Web Scraper API.
Gate 2 - build it. No ready scraper covers these fields, and any one of these: many pages share one layout, the job will run again, or the data only appears after browser actions (typing, scrolling, waiting, CAPTCHA) and the user has no Playwright, Puppeteer or Selenium setup.	Scraper Studio.
Fall-through. One-time job, no ready scraper, and the pages share no layout.	Back to fetch (Web Unlocker), and the user owns the parser.

Many pages means many URLs. One page holding many items is still one page, and with no ready scraper and no recurrence it falls through to fetch, which is a normal outcome, not a failure.

Gate 1 in practice. Open references/web-scraper-api.md and check its bundled top-25 table first, which costs zero API calls. When nothing matches, run node scripts/find-scraper.mjs <query> with a gd_ id, a domain, or part of a scraper name - free reads of the scrapers catalogue (GET /datasets/v3/scrapers, filters ?dataset_id= and ?domain=), junk rows skipped. An id prints every variant's typed inputs, output fields (first 8, all with --json) and trigger line on its own, --sample adds the sample input, a name or domain needs --schema for the same, and --variant <door> narrows it to one door. Manual fallbacks for the same lookup: bdata pipelines search "<site or data type>", or curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" "https://api.brightdata.com/datasets/v3/scrapers?domain=<site>". All are free reads that trigger no job. An id the user or a colleague handed over gets the same treatment - resolve it with the script before triggering, because a handed id can be a marketplace row, stale, or the wrong variant. With the CLI, any scraper in the catalogue runs by id: bdata pipelines run <dataset_id> <method> "<url>". An id the CLI rejects as unknown runs only through REST or scripts/trigger.mjs. Without the CLI, node scripts/trigger.mjs <dataset_id> "<url>" starts the job and node scripts/poll.mjs <snapshot_id> brings back the records. One URL can return many records (a profile's posts, a product's reviews), each one billed. --limit <n> is sent as limit_per_input, documented for discovery requests, so check the record count of the first run. Run the check and pick the scraper yourself, rather than asking the user which one to use. A single URL is not a reason to skip it. Web Scraper API has no built-in scheduler, so "daily" means the agent writes a cron job or a GitHub Actions workflow in the user's project.

When the user has no URLs. Many ready scrapers take a discovery input instead: a keyword, a category URL, a best-sellers URL, a location. Read the scraper's catalogue entry, its input_schema and discovery doors, before assuming it needs a URL; metadata lists outputs only. Sites with no ready scraper go to Scraper Studio, which covers single-page, search-discovery and multi-page patterns.

Gate 2 in practice. Scraper Studio has a built-in scheduler (hourly, daily, weekly, or custom). It heals itself when the site changes, and a heal can wait for the user's approval (scraper heal and scraper approve in the CLI). It also handles multi-page crawl jobs, and it does the browser work for you. For an urgent ask on a collector that already exists, bdata scraper run <collector_id> <url> --sync returns data in one call, single URL only, server-side cap of 25 to 50 seconds. If it times out it prints a response_id: poll GET /dca/get_result?response_id= with it rather than re-running, which starts a second run. For many URLs at once drop --sync: --urls a,b,c or --input-file urls.txt sends one batch through /dca/trigger and polls until the records land.

The boundaries

Instagram profiles, posts, reels and comments belong to the instagram skill, which carries their dataset ids and quirks. Pages themselves belong to fetch: markdown, HTML, a screenshot, "for my RAG", or an explicit hurry word ("now", "quick", "ASAP") on a one-off pull. A single URL is not a hurry word, and a live chat is not a hurry word. Anything starting from a search query belongs to search. A user who already drives a browser with Playwright, Puppeteer, Selenium or a computer-use model belongs to the browser skill. A user with no such setup stays here, and Studio does the browser work.

The two override words

• fields (or "records") - forces structured data fields. Run gate 1, then gate 2.
• recurring - puts the job on a schedule. It does not skip the gates. With a ready scraper the product stays Web Scraper API and the agent writes the cron. With no ready scraper, "runs again" is what sends the job to Studio.

Two separate words, because output and schedule are two separate choices.

Web Scraper API is asynchronous by default

Trigger, poll, download. A synchronous POST /datasets/v3/scrape exists for small jobs, but past one minute it also hands back a snapshot_id, so never promise an instant answer. bdata pipelines run does the polling, so the command looks synchronous, but the job underneath still takes its time.

Example - one TikTok profile

The ask: "get me the follower count and bio for tiktok.com/@nasa"

1. Fields, not the page. Stay in this skill.
2. One specific input, not a generic corpus. No Marketplace.
3. The top-25 table has TikTok profiles (gd_l1villgoiiidt09ci). Gate 1 wins. One URL still gets the library check.

bdata pipelines run gd_l1villgoiiidt09ci collect_by_url https://www.tiktok.com/@nasa

Then state the choice in one line while the job runs:

Using the maintained TikTok profiles scraper. It runs as a job, so this is not instant. Say recurring to put it on a schedule.

When the records come back, check they describe the entity the user meant before reporting them. Vanity URLs collide, and linkedin.com/company/anthropic returns a venture fund rather than the AI lab.

Read next

• Before the first Web Scraper API call: references/web-scraper-api.md - the top-25 table with each scraper's discovery doors, the name trap, and the free discovery moves.
• Before scraping anything that sounds like a huge generic corpus: the datasets skill - the three-part gate, the free coverage checks, and why the agent checks and routes but never buys.
• When gate 2 fires and you are about to build a scraper: references/scraper-studio.md - the four CLI commands that exist, the create-then-run split, the approval gate on a heal, and the browser work you no longer have to write.
• When every gate has said no and the data job has to fetch raw pages itself: references/web-unlocker.md - the one-POST request shape, the four refusal codes (three clear with KYC), and the signal that means go back to gate 2.
• The moment a trigger hands back an id instead of data: references/snapshots-and-jobs.md - which endpoints match which id, and the delivery settings the agent never configures.
• When a bundled id is rejected or the site is not in the table: scripts/find-scraper.mjs - free reads of the scrapers catalogue by gd_ id, domain or name; an id prints every variant's inputs, outputs and trigger line, a name or domain adds --schema for them, --variant picks one door, one empty-body probe only for an id the catalogue omits (required fields certain, outputs unlisted; on a door with no required field it can start a billed job, and the script then prints the snapshot id and the cancel call), junk rows filtered, key never printed.
• When the machine has no CLI: scripts/trigger.mjs - node scripts/trigger.mjs <dataset_id> "<url>" --limit <n> starts the job (one URL, but it can return many billed records; --limit is sent as limit_per_input, documented for discovery requests) and prints the snapshot id, key never printed. After an error, read it before running again: a timeout or a 5xx can come after the job started.
• When a job is running and you need the data without hand-writing a poll loop: scripts/poll.mjs - either id in, data out, key never printed.
• When a call is refused for auth: the agent-onboarding skill. A REST 401 that says the key is missing or wrong (Credentials are missing, User authentication is required, Invalid credentials, Invalid token) and a CLI line starting "Error: No API key found." all mean log in. Auth method is not supported and Credentials are invalid mean the header itself is wrong: send exactly Authorization: Bearer <key>. Send 407 and kyc_required there too, along with missing zones.

Red flags - stop if you catch yourself doing one of these

• Skipping the library check because there is only one URL
• Hardcoding a guessed dataset_id instead of using the table or a live search
• Reaching for Web Unlocker while a ready scraper covers the fields