In March 2026, consumer authorities from 23 EU countries plus Iceland and Norway published their check of Black Friday and Cyber Monday sales: 30% of the 314 online shops they checked referenced discounts incorrectly. The rule those shops broke fits in one sentence. Checking it takes data you should have started collecting a month before the sale.
The Challenge
Under the EU's Price Indication Directive, a shop that announces a price reduction has to quote, as its reference, the lowest price it applied in the previous 30 days. The Omnibus Directive added that rule in 2019, and shops have had to follow it since 2022. Black Friday falls on 27 November this year, so the window that decides whether a Black Friday "was" price is honest opened on 28 October. A tracker that starts when the banners go up is a month late.
The Commission's own monitoring, built with Apify, covered 16,000 products in 176 shops for the 2022 Black Friday sweep and by October 2025 had helped check 136,931 products from 1,556 retailers. It collected prices during the sale and the 30 days before, and kept a screenshot of each page as evidence. Oxylabs ran a US version in 2024: it tracked 61,172 products every hour from 13 August, and only 4,529 of them were both on sale on Black Friday and listed for the 90 days its method required. Product lists churn faster than most people plan for.
But the hard part isn't scale. It's the statistic.
A reference price is a minimum, and a minimum is decided by a single observation. An average shrugs off one bad day; a minimum is that day. That's why both ways a scraper goes wrong flip the verdict:
- A missed day hides the low. The page answers with a challenge page, a consent screen or a timeout on the one day the price dipped. Your minimum comes out higher than the truth, and a fake discount passes as real.
- A misread price invents a low. Your parser grabs a "from" price, a member price, another variant or an accessory in a carousel. Your minimum comes out lower than the truth, and an honest discount gets flagged as fake.
Dips can be short. In its 2025 Black Friday research, Which? followed 175 products at eight UK retailers for a year: 83% were the same price or cheaper at least once outside the four-week sale, and one TV it tracked briefly dropped from £1,039 to £999 the day after Black Friday. (The UK has its own rules, but a one-day dip looks the same to a scraper.)
For a deal site, the first error costs credibility. For a regulator, a consumer group or a newsroom, the second one means accusing a retailer of something it didn't do.
The Approach: A Gap Stays a Gap
You need a collection layer that never turns a failed fetch into a number, and a calculation that refuses to give a verdict over a window with holes in it. We covered how a bad fetch becomes a fake signal in Alternative Data: When a Block Looks Like a Data Point; a 30-day minimum is the least forgiving version of that problem.
With a platform like FourA, the moving parts are small:
- Prove the page before you read it.
validatetakes a string only the right product page carries, such as the item's own article number. A challenge page, a "product not found" page or a consent screen comes back as a failed call, which isn't billed because your rule decides what counts as success. Your row gets a gap, not a price. - Keep the receipt. Responses carry an
X-FourA-Request-Id. Store it with the price, the timestamp and the raw HTML. Our Activity log keeps request previews only 24 hours, so the evidence archive has to live on your side. - Pick the endpoint per shop. Single handles server-rendered product pages. Browser is for shops that build the price in JavaScript. Proxy Finder with
exitCountries(where your plan includes country targeting) covers shops whose price depends on the visitor's country, a cousin of the problem we met with store-level grocery prices.
import datetime as dt
import requests
API = "https://eu.api.foura.ai/api/single/"
HEADERS = {"X-API-Key": "pk_live_..."}
def observe(product, parse_price):
"""One observation. A failed fetch is a gap, never a price."""
r = requests.post(API, headers=HEADERS, json={
"method": "GET",
"url": product["url"],
"validate": {
"status": {"accept": [200]},
"data": {"accept": [product["article_no"]]},
},
})
body = r.json()
row = {"day": dt.date.today(), "url": product["url"], "price": None,
"request_id": r.headers.get("X-FourA-Request-Id")}
if r.status_code == 200 and not body.get("error"):
row["price"] = parse_price(body["data"]) # None when it isn't sure
row["html"] = body["data"] # the evidence
return row
def prior_price(rows, sale_day, days=30):
start = sale_day - dt.timedelta(days=days)
priced = [x for x in rows
if start <= x["day"] < sale_day and x["price"] is not None]
if len({x["day"] for x in priced}) < days:
return None # a missing day can hide the lowest price
return min(x["price"] for x in priced)
prior_price returning None is the most useful line in the file. A discount you can't verify goes on a "no verdict" list, with the missing days named. We'd rather ship a report with that list than one that's confidently wrong.
Two more decisions belong before the sale, not during it. First, sample more than once a day on categories that move fast; Oxylabs looked every hour for a reason. Second, collect further back than 30 days. The rule looks back exactly that far, so a price raised just before the window opens and held until the sale is invisible to it. Whether that's misleading is a question for the Unfair Commercial Practices Directive and a lawyer. Whether you can show it happened is a data question, and only a longer history answers it.
Results
Illustrative scenario based on the setup above: a consumer group checking 2,000 products across 40 shops in three countries, twice a day, from 1 October to Cyber Monday.
- 244,000 observations, each stored with its request id and raw page.
- Zero challenge pages in the price column. Every row either passed its product check or is an explicit gap.
- Three verdicts, not two. A discount is a breach, compliant or "no verdict", and the third one lists the days it's missing.
- Evidence attached. Every flagged deal ships with the timestamps and pages behind its 30-day minimum.
- Pre-window hikes in view. Eight weeks of history shows a price that went up just before 28 October and "dropped" on 27 November.
- Free gaps. A rejected fetch isn't billed, so a same-day retry costs only the call that works.
Key Takeaway
Most scraping pipelines are built to be right on average, and averages forgive. The 30-day rule asks for the one number a pipeline is worst at: the extreme. One invented low or one lost dip, and the verdict flips.
So the bar for a discount checker isn't "mostly right". It's "never makes up a low, never loses one, and says so when it can't tell". The window for the January sales opens before Christmas, which means the clock for the next verdict is already running.