Web scraping without getting blocked is not one problem. It is a stack of them, and each layer leaves a different signature in your logs. A 403 on the first request means something different from a 403 on the four-hundredth, which means something different again from an HTML challenge page where JSON used to be.
Match the symptom you actually see, start at the top, and stop when the logs go quiet.
Everything below assumes legitimate data access: public pages, nothing behind an authentication wall, and a request rate the target can absorb. The list is diagnostic. Every fix on it makes your client better behaved, not better disguised.
1. Every request is refused, including the first one
Your client is not pretending to be a browser, and it is not trying very hard. The default header set from a scripting HTTP library gives it away twice over, once for what it contains and once for what it leaves out.
GET / HTTP/1.1
Host: example.com
User-Agent: python-requests/2.32.3
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-aliveA current Chrome sends a dozen or more headers in a stable order, including Accept-Language, Sec-Fetch-Site, Sec-Fetch-Mode and the Sec-CH-UA client hints. Order matters as much as content. A header set that is alphabetised, or that carries Chrome’s user-agent without Chrome’s client hints, contradicts itself.
The fix is to copy a real request out of your own browser’s devtools and replay it exactly, order included. It costs nothing, and it buys you very little against a defended target. This is the fix that stops working the moment anyone inspects the layer below HTTP.
2. Headers are perfect and you are still blocked before any JavaScript runs
This is TLS fingerprinting. Your handshake gets inspected before a single byte of that carefully constructed header set is read. JA3, published by John Althouse, Jeff Atkinson and Josh Atkins at Salesforce in 2017, hashes five fields from the TLS Client Hello into a 32-character identifier.
SSLVersion, Ciphers, Extensions, EllipticCurves, ECPointFormats
-> joined with commas and hyphens, then MD5
771,4865-4866-4867-...,0-23-65281-10-11-...,29-23-24,0Cloudflare’s bot documentation says JA3 and JA4 fingerprints “identify TLS clients based on how they initiate connections” and act as “a stable identifier across different destination IPs, ports, and certificates”. JA4, the successor, sorts the extension list first, which collapses the noise Chrome introduced by shuffling extensions.
None of this is set by your code. It comes from OpenSSL, or BoringSSL, or Go’s crypto/tls. A Python client claiming to be Chrome 140 produces a handshake no Chrome has ever produced.
The fix is an impersonating client such as curl-impersonate, or a real browser engine. Impersonation libraries track browser releases and always lag them, so the cost is a maintenance treadmill that runs as long as your scraper does.
3. HTTP/1.1 works and HTTP/2 does not
The HTTP/2 connection preface is itself a fingerprint. Elad Shuster’s Akamai white paper, presented at Black Hat Europe in 2017, derived HTTP/2 client fingerprints from more than 10 million HTTP/2 connections collected across Akamai edge servers.
The format joins four sections with pipes: the SETTINGS parameters in the order sent, the connection-level WINDOW_UPDATE increment, the PRIORITY frame contents, and the order in which the client emits its :method, :authority, :scheme and :path pseudo-headers. Your HTTP/2 library picks all four. You pick none of them.
So the fix is the one above, a layer deeper: use a client whose h2 implementation is a browser’s, which in practice means driving the browser. Very few targets check this. If you are hitting it, you are scraping something well defended and the cheap fixes are already behind you.
4. Fine for a few hundred requests, then a wall
You are being rate limited. This is the most common reason a working scraper stops working, and the signature is a clean break at a round number of requests, or at a consistent requests-per-second ceiling.
Look for HTTP 429, defined in RFC 6585 by Mark Nottingham and Roy Fielding in April 2012 as “the user has sent too many requests in a given amount of time”. Behind Cloudflare the same condition surfaces as error 1015.
Honour the Retry-After header, which RFC 6585 explicitly provides for. Cap concurrency per host rather than globally. Back off exponentially on 429 and 503, and read a sustained 429 rate as a signal to lower your ceiling permanently rather than to retry harder.
The cost here is throughput, and only throughput. No amount of fingerprinting work fixes a rate problem, and plenty of teams buy proxy infrastructure when a token bucket would have done the job for free.
5. You are blocked on paths robots.txt already told you about
You are crawling what the operator asked you not to crawl. RFC 9309, published in September 2022 by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, standardised the Robots Exclusion Protocol, and it is more specific than most implementations assume.
- A cached robots.txt SHOULD NOT be used for more than 24 hours.
- If robots.txt returns a status in the 500 to 599 range, the crawler MUST assume complete disallow.
- Parsers must handle at least 500 kibibytes of the file.
Parse robots.txt properly, refresh it daily, and honour crawl-delay where it is set. Some paths become off-limits, which is the correct outcome. It is also cheap insurance, because an operator who can see you obeying the rules has less reason to escalate.
6. You get a challenge page instead of your data
The site cannot tell what you are, so it is asking. Cloudflare describes a Managed Challenge as asking “the browser to perform a series of checks that help confirm the visitor’s legitimacy” by evaluating client-side signals, and says plainly that it “does not use CAPTCHA puzzles or visual tests”.
There are two honest answers. If your crawler serves a public purpose and can hold still, apply to Cloudflare’s Verified Bots programme. It asks you to identify yourself through Web Bot Auth signatures, a published IP list with a stable user-agent, or reverse DNS, and to obey robots.txt while keeping request rates reasonable.
Verification costs you IP churn and user-agent rotation, permanently, and for a lot of crawlers that is the right trade. If it does not fit the workload, the alternative is to run the traffic somewhere the client-side signals are real rather than synthesised.
7. It works on your laptop and dies in CI
The network you moved it to is the problem. Datacenter IP ranges are identifiable by ASN in a single lookup, and a headless browser on a cloud VM has to synthesise hardware it does not have: a GPU, sensors, touch input, a radio. Every synthesised value is somewhere the approximation can diverge from what real hardware reports.
Routing that VM through a residential proxy narrows the gap without closing it, because the browser and the exit IP still sit on different machines with different characteristics.
In increasing order of cost, the options are residential proxies, then real-device execution. Archonum runs the browser on the phone that owns the IP: 250,000+ real consumer smartphones across 175+ countries, one network hop, drop-in for existing Playwright and Puppeteer scripts. Because the device, the browser and the IP are the same physical handset, there is no synthetic signal to detect in the first place.
Real devices are the most expensive tier here, and they are overkill against a target that only checks ASN. If a residential proxy clears your block rate, use the residential proxy.
8. It worked for a week, then broke at 2am
You are not measuring block rate, so a downstream consumer told you instead. Detection vendors ship changes continuously, and a configuration that worked last Tuesday carries no guarantee about this one.
Log the status-code distribution per target host. Alert on the ratio of non-200 responses rather than on absolute failures, and keep a canary that fetches one known-good URL per target on a schedule. Give block rate a number and treat it as a service level objective.
This costs engineering time and ships no features. It is also the only item on the list that tells you which of the other seven you are currently failing.
Most blocked scrapers are stuck at item 4. Work down the list in order, fix the cheapest cause that matches the symptom, and escalate the infrastructure only when the logs prove you have to.