Market Research Web Scraping on Real Devices

Sampling integrity is the real constraint on market research web scraping

A collection run that finishes is not the same as a collection run that represents the market. Web-sourced research datasets are usually judged on record count, and record count is the one property that tells you nothing about whether the sample is any good.

The failure mode is quiet. A collector works in Germany and the United States, gets challenged in Brazil, and is served a stripped-down page in Indonesia. Nothing throws an error. The export still lands on schedule. What you now hold is a dataset describing the markets where collection happened to be easy.

Blocking is a sampling problem before it is an uptime problem

When bot detection responds differently by region, the records you lose are not lost at random. They correlate with the variables research cares about most: country, carrier, device class, and frequently price tier. In Donald Rubin’s 1976 classification of missing data mechanisms, this is the missing-not-at-random case, the one you cannot repair by reweighting afterwards, because the reason a record is absent is bound up with its value.

Engineering reports this as a success rate. Research needs it as coverage: which strata are complete, which are thin, which are empty. The two readings of the same run disagree more often than they agree. If you want the mechanisms underneath, the diagnostic walkthrough is in web scraping without getting blocked.

Panel data and scraped public data answer different questions

A consumer panel measures people: what a recruited and weighted group of individuals reported, bought, or viewed, with known demographics and stated error bars. Scraped public data measures pages: what a site displayed to a visitor from a particular place at a particular moment, with far more breadth, much fresher timestamps, and no demographic frame at all.

Neither substitutes for the other. Panels answer who. Public web data answers what was on offer. Trouble starts when a deck presents scraped listings as consumer behaviour, or treats a competitor’s published price as a price somebody actually paid.

Longitudinal series break when the vantage point moves

Trend work assumes the instrument holds still. In web collection the instrument is the vantage point: the exit IP, the device, the browser, and the location a site infers from all three. Change it halfway through a study and a step in your chart may record nothing more than the site switching you to a different page variant.

Three habits keep a series honest:

• Fix the vantage point per market: same country, same network type, same device class, for the life of the study.
• Store the collection context alongside every observation, so a later analyst can tell a market movement from an instrument change.
• When a vantage point has to change, annotate it as a break in the series rather than reading the shift as signal.

Where real devices change the picture

Archonum runs AI-agent browser sessions on a network of 250,000+ real consumer smartphones across 175+ countries. The browser runs on the handset that owns the residential IP, so there is no proxy split between the request and the exit address, and nothing synthetic for bot detectors to flag.

For research work the consequences are narrow and specific. Coverage stops depending on which markets a provider serves cleanly, because sessions originate inside the market being measured rather than being routed toward it. A geo-pinned session can be held stable across months, which is what a longitudinal price or assortment series requires. Existing Playwright or Puppeteer collectors run unchanged, so the sampling design does not bend around a new SDK. The trade-offs against proxy-based collection are laid out in datacenter vs residential proxies vs real devices, and a worked example sits in the price monitoring use case.

The boundary we work inside

Archonum is data-access infrastructure for publicly available information: catalogue pages, listed prices, public reviews, published availability. It is not a source of personal data, it does not supply accounts or credentials, and it is not built to reach anything behind a login. Research that needs individual-level consumer data belongs in a consented panel, which is a different instrument with a different governance model.