Web Scraping Guides

What every web scraping guide here has in common

Each web scraping guide on this page follows one target end to end: what the page actually serves, how the site behaves once requests stop looking like a single curious human, and where the extraction breaks at volume. The order is the same every time. Inspect the response, build the smallest thing that works, then find out what fails when you run it for a week.

The failure modes get more space than the happy path. A tutorial that stops at a working selector is easy to write and useless by the second week, when the selector still matches but the responses have quietly become challenge pages.

None of them are language tours. If you can read a CSS selector and an HTTP status code, you have enough to follow any of them, in whatever stack you already use.

Choosing the one you need

Pick by constraint rather than by industry. Two sites in unrelated markets often share a defence vendor, which makes their guides more transferable to each other than to a competitor running plain rate limits.

• If you have never shipped a scraper, start with the target you already know as a user. Familiar page structure removes half the debugging.
• If the problem is parsing, any of them will do. If the problem is being let through at all, read the guide for the site closest to yours in defence posture.
• If the output feeds a model rather than a dashboard, start with the pipeline guide. Schema and provenance decisions are expensive to reverse after a few million rows.
• If you are already being blocked in production, the diagnostic in Web Scraping Without Getting Blocked is faster than any per-site walkthrough.

What to have in place before you follow one

Every guide assumes some groundwork that is not worth repeating in five places.

• A written list of the fields you need and what each one is for. Undefined scope turns into request volume nobody budgeted for.
• A read of the target’s robots.txt and terms, and a decision about your use of the data that you would be comfortable explaining out loud.
• Retry, backoff, and concurrency limits chosen before the first run, not added after the first outage.
• Logging that separates “the markup changed” from “we were refused”. Both surface as empty results, and confusing them costs days.
• Somewhere durable to write the data, with the fetch timestamp and source URL kept alongside every record.

Where a guide stops and infrastructure starts

Every guide here eventually reaches the same boundary. Parsing a page is a bounded problem you can learn in an afternoon. Still being trusted by that site in six months is not, and no amount of code quality fixes it, because the decision is made before your parsing code runs.

What a detector reads first is where the request came from and what the client claims to be. A headless browser on a cloud host is identifiable from its IP range, its TLS handshake, and a long list of fingerprint surfaces that no header override reaches. Browser fingerprinting layer by layer covers what those surfaces expose, and datacenter vs residential proxies vs real devices covers what the network layer gives away.

Archonum closes that gap instead of disguising it. AI agents run their sessions on a network of 250,000+ real consumer smartphones across 175+ countries, so the browser and the residential IP belong to the same handset. There is no proxy split and nothing synthetic for bot detection to flag. The guides still apply; the part they cannot teach is handled underneath them.

New guides are added as targets and defences change. Longer analysis pieces live on the Archonum blog.