E-Commerce Data Scraping for Competitor Price Monitoring

Why e-commerce data scraping fails quietly

Most e-commerce data scraping does not fail with a 403. It fails with a 200 and the wrong number inside it.

Retail and marketplace sites decide what to show you before they decide whether to block you. Currency, tax treatment, delivery estimate, in-stock flag, promotional badge, and which seller wins the buy box are all resolved against the visitor’s country, and often against the fulfilment region behind it. A request from a datacenter range in the wrong country still gets a page. It is simply not the page a shopper in that market sees.

This is why “the crawl completed” is a weak success metric. A job can report near-total coverage and still feed a repricing model numbers from a storefront none of your customers shop on. The error is silent, and it is directional rather than random, so it survives every sanity check that only looks for gaps.

Geography and device class change the answer, not just the access

Two variables move retail output more than any anti-bot control does.

• Location. Large marketplaces run separate country storefronts with separate catalogues, sellers, and prices. Inside a single storefront, delivery windows and same-day availability still resolve against a postcode.
• Device class. Retailers serve mobile browsers different templates, different pagination, and sometimes different app-linked promotions than a desktop session gets. A desktop-shaped crawler collects the desktop truth, which may not be the one most of your customers act on.

Neither of these is a defence mechanism. Both are ordinary commerce. But both mean that a collection stack which cannot present as an actual consumer in a given country, on a given class of device, returns data that is accurate about a market you did not ask about.

Refresh cadence is the real constraint

Competitor price monitoring is a cadence problem before it is a parsing problem. A long-tail catalogue is well served by a daily pass. A marketplace listing where the buy box changes hands through the day is not usefully sampled once every 24 hours.

Cadence is where most stacks hit their ceiling. Shorten the interval and challenge rate, block rate, and cost per record all climb together, so teams trade freshness for reliability by quietly cutting SKUs from the list. The way out is to lower the risk each session carries, not to push more requests through a channel that is already flagged.

Archonum runs those sessions on real consumer smartphones: 250,000+ devices across 175+ countries, with Chrome running on the phone that owns the residential IP. There is no proxy split between browser and exit node for a detector to notice, and no emulator signature to flag, because nothing in the session is synthetic. Geography is not spoofed either; it is where the handset actually is. For the diagnostic sequence when collection starts degrading, see web scraping without getting blocked, and datacenter vs residential proxies for how the network options compare.

Where the line sits

Public product data only: prices, titles, specifications, ratings, and availability that any shopper can see without signing in.

The Ninth Circuit’s 2022 decision in hiQ Labs v. LinkedIn held that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act. It did not settle everything. The same dispute went on to turn on LinkedIn’s user agreement, which is a contract question rather than a hacking one.

The practical boundary is narrow and worth writing down: no account creation, no logged-in sessions, no circumvention of an access control, and request volume a site owner would recognise as ordinary traffic. Copyright and database rights still attach to the content itself, and anything personal, such as reviewer names and profiles, carries obligations under GDPR and its equivalents. Archonum supplies the device and browser layer for your agents; it does not supply accounts or credentials, and it is not built for credentialed access.

None of that requires pretending to be a customer. The traffic passes because it is genuinely human traffic, generated by a real browser on a real phone, and that is a different claim from a good disguise.