Web Scraping for AI Training Data

Web scraping for AI training is a dataset-quality problem

A corpus fails quietly. When a collection request is challenged, geo-redirected, or handed a consent wall, what comes back is often still well-formed HTML: it parses, it tokenises, and it lands in the dataset looking like every other document. Nothing throws. The damage surfaces weeks later, during evaluation, when a slice of the training set turns out to describe interstitial pages rather than the domain you meant to sample.

This is the part of web scraping for AI training that infrastructure actually decides. Parsing, deduplication, and labelling are downstream problems with mature tooling. Whether the bytes reaching that tooling are the page a real user would have seen is a collection problem, and it is settled before the first parser runs.

Four ways LLM training data goes wrong at collection time

• Silent substitution. A challenge page, a login wall, or a “we could not verify your browser” interstitial is valid markup. A status-code check misses it. Length and content assertions on every fetch catch more of it than retry logic does.
• Geographic and language skew. A corpus inherits the vantage points it was collected from. Regional pricing, localised copy, and non-English variants go missing not because anyone excluded them, but because those requests failed more often than the ones that succeeded.
• Near-duplicate inflation. Retries, pagination loops, and syndicated content multiply the same text. Katherine Lee and colleagues reported in Deduplicating Training Data Makes Language Models Better (2021) that removing duplicated sequences reduced verbatim memorisation and let models reach the same accuracy in fewer training steps.
• Missing client-side state. Pages that assemble themselves after hydration return a shell to a plain HTTP fetch. An evaluation set built that way tests the model against markup no human ever read.

What real devices change about collection

Archonum runs agent browser sessions on 250,000+ real consumer smartphones across 175+ countries. The browser executes on the phone that owns the residential IP, so there is no proxy split between the request and the exit, and nothing synthetic for bot detectors to flag.

For a training pipeline the practical consequence is coverage that behaves the same way twice. Geographic and language spread becomes a sampling decision you make deliberately, rather than an artefact of which vantage points happened to survive the week. Pages render the way a real user’s mobile browser renders them, which matters most for evaluation sets meant to reflect real reading conditions.

Existing Playwright and Puppeteer jobs point at it without a rewrite. The diagnostic sequence in why scrapers get blocked and the datacenter, residential, and real-device comparison cover where each of the cheaper options stops holding.

Provenance, robots.txt, and the boundary

Record where every document came from: URL, fetch timestamp, HTTP status, the country the session ran from, and the render mode. Provenance is what lets you re-derive a slice, honour a later removal request, or answer a question about origin without re-crawling the web.

The scope is publicly accessible content, collected with respect for robots.txt — a standards-track protocol since RFC 9309 (Koster, Illyes, Zeller and Sassman, IETF, September 2022) — and for site terms of service. No paywalled material, no account-gated content, no personal data.

Two honest limits. Filtering carries a bias of its own: Jesse Dodge and colleagues, in their 2021 audit of the C4 corpus, found that its cleaning filters removed text associated with minority dialects more often than baseline, so a clean pipeline is not automatically a representative one. And licensing and copyright for training data stay with the collector. Infrastructure can tell you exactly what was fetched, from where, and when. It cannot tell you whether you are permitted to train on it. The pipeline guide below walks the collection layer end to end.