Web Scraping Resources

Where web scraping techniques are actually documented

Most published material on web scraping techniques is vendor copy citing other vendor copy. The primary sources are older, shorter, and considerably more precise, and there are far fewer of them than the volume of blog posts suggests.

Four are worth reading before anything else on this page.

• RFC 9309, Robots Exclusion Protocol (IETF, 2022). The first standards-track definition of robots.txt after roughly 25 years of informal convention. It specifies parsing rules and caching behaviour that most crawler implementations get wrong.
• RFC 6585 (Nottingham and Fielding, April 2012). Defines 429 Too Many Requests. Treat it as a scheduling instruction from the origin rather than a failure to retry through.
• Mitigating Browser Fingerprinting in Web Specifications (W3C). Written for specification authors, so it enumerates the fingerprinting surfaces without selling a product at the end.
• Browser Fingerprinting: A Survey (Laperdrix et al., 2019). The academic baseline that most vendor explainers are quietly paraphrasing.

Vendor engineering blogs are the one commercial category worth tracking, and the useful ones sit on the detection side rather than the scraping side. A defender’s own account of what it measures carries more information than a third party’s summary of it.

How to tell a maintained reference from a stale one

This subject rots quickly. A guide written against a 2021 browser build describes an engine that no longer ships, and the automation stack around it has moved several times since.

• Does it name versions and dates? A fingerprinting reference that never mentions a specific Chrome version is describing a browser in the abstract.
• Are the claims falsifiable? “Undetectable” is not a testable property. “Passes this named check as of this date” is.
• Which direction do the citations run? Down to a specification or a paper is a good sign. Sideways to three other blog posts is not.
• Does the code still run? API surfaces such as Playwright’s BrowserContext change often enough that an unmaintained tutorial fails on the first import.

The dated, versioned, slightly boring source usually beats the confident one.

How the material on this site is organised

This archive holds reference lists and market overviews, the material you read once to get oriented rather than the material you keep open while debugging. The rest of the site is split by the question you arrived with.

Where to go next

• Guides and tools for implementation detail, comparisons when you are choosing between approaches, and use cases when you are checking whether a workload is a fit.
• Two posts on the blog carry most of the technical weight: browser fingerprinting layer by layer, which maps what each tier of the stack reveals, and web scraping without getting blocked, which works through eight failure modes from symptom to fix.

What this library deliberately leaves out

There are no evasion recipes here, and no material on defeating a specific security control. That is a positioning choice rather than a legal disclaimer.

Archonum’s position is that the detection problem is not solved by better spoofing. Agent traffic on our network runs on 250,000+ real consumer smartphones across 175+ countries, on genuine mobile browsers and genuine residential IPs, which means there is nothing synthetic for a bot detector to flag in the first place. That framing decides what gets published: material explaining how detection works, not material helping anyone impersonate a person who does not exist.

Best practice in this field is mostly restraint. Respect the robots file, honour 429 and Retry-After, request only what you need, and identify yourself where the site expects it. Most of what teams describe as an anti-bot problem starts as a request-rate problem.