The EDPB Just Wrote a Spec for Scrapers
Europe's data protection authorities have said a lot about AI models. Until this summer, none of it told an engineer what a scraper should do before it fetches its first page.
Guidelines 03/2026 on web scraping in the context of generative AI changed that. The European Data Protection Board adopted the draft on 7 July 2026, and its public consultation closes on 30 October 2026. On paper the scope is narrow: private companies scraping personal data to train or fine-tune generative AI. In practice, it's the most detailed description a regulator has published of how a defensible scraping job is built, from the criteria you write before the first request to the timestamp on the last record.
Our read, after going through all 23 pages: the draft doesn't care which tools you use. It cares whether you can say what you collected, from where, and when. Scrapers that can answer that in one query will be fine. Open-ended crawls will struggle with almost every paragraph.
(Disclaimer: This is not a legal advice. Firstly, we're referring to a draft, and we're engineers reading it for what it means from the perspective of code.)
Targeted or untargeted is the line that matters
Paragraph 8 splits scraping into two kinds by how specific and restrictive the collection is, "and not the technical tools used". Targeted scraping follows collection criteria: a domain, a language, a topic. Untargeted scraping is a crawler that follows every link it finds. The Board's worry is specific: as the queue grows, the controller "has limited knowledge of what personal data are collected and processed."
That split runs through the rest of the text. Data minimisation asks you to "define precise collection criteria" and run a "data mapping and inventory process" before collection starts (paragraph 37). The necessity test under legitimate interest says narrowing the criteria, "rather than scraping a wide part of the internet", may be what makes the processing necessary at all (paragraph 52). Transparency asks for a published list of sources, ideally with domain names, URLs and collection dates (paragraphs 30 and 31).
A price tracker with a fixed list of product pages on a dozen retail domains can produce every one of those artifacts from its config file. A breadth-first crawl can't. Privacy World's critique of the draft names the tension well: the obligations are written for a controller who knows what it collects, right after the Board concedes that an untargeted crawl may hold personal data it can't inventory.
What this means: if you scrape anything that touches people (reviews with usernames, listings with an agent's name and phone number, forum threads), your collection criteria are now a compliance artifact. Keep them in version control next to the scraper, not in someone's head.
A site's "no" now counts against you
This is the part that will change the most pipelines. The draft lists the technical measures that show a site clearly opposes being scraped: robots.txt, ai.txt, "robots authentication to view content", and checks that "impose an action that can only be carried out by a human being" (paragraph 37). Its advice for training data is to leave those sites out.
The same signals come back in the balancing test. When people know a site uses those measures and it gets scraped anyway, the draft says they're less likely to expect the processing (paragraph 64). Example 5 spells it out: a platform with a robots.txt exclusion, a human-only check and a stated ban on AI training means its users "cannot reasonably expect" third parties to scrape their data for that purpose. And the reverse doesn't hold. A missing robots.txt isn't consent (paragraph 45).
France's CNIL got there first. Its focus sheet on legitimate interest and web scraping, first published in June 2025, says processing "cannot fall within the reasonable expectations of data subjects" when the controller doesn't exclude sites that object through robots.txt or a human-only check.
Two less obvious items sit among the mitigating measures (paragraph 66c). One excludes personal data that's only visible after logging in. The other covers sites that return different output to a scraper's user agent than to a regular browser, "particularly where the scraping service receives more data" than a person would see. If your scraper reads a richer internal API than the page shows, that's the case they mean.
We looked at the machine-readable side of this (robots.txt extensions, signed agents, content signals) in Search, Agent, Training: The Web's New Bot Rules, and at publishers who'd rather charge than refuse in Pay-Per-Crawl Is Splitting the Web in Two. The EDPB draft is the first time a regulator has treated those signals as evidence of what people expect.
What this means: record the state of robots.txt (and ai.txt, where a site has one) per domain at collection time, next to the data. If someone asks in a year why a source ended up in a training set, "it was allowed on the day we fetched it" is only an answer if you wrote it down.
The scraper may be the processor
Section 3.2.1 is short and easy to skip. It shouldn't be, especially if you build scrapers for clients. If you create a training dataset on the developer's behalf "according to its documented instructions", you may be a processor and the developer the controller (paragraph 18). If you and the client set the collection criteria together, you may be joint controllers (paragraph 20, Example 1). If you scraped a dataset on your own and someone else reuses it, each of you answers for your own processing (paragraph 19).
Brian Hengesbaugh's IAPP column on the draft makes the commercial point. When a contract names the vendor as processor, the client carries the compliance work the guidance describes, and the vendor's own direct duties "might be more narrow and less comforting than otherwise thought." That cuts both ways. A client can't hand compliance to the freelancer who wrote the scraper, and a freelancer who improvises the source list may end up a controller by accident.
What Data Teams Should Change Now
Most commercial scraping isn't touched by this. Prices, stock levels, product specs and timetables generally aren't personal data, so the GDPR doesn't reach them. The draft is explicit that in a "mixed" dataset it applies to the personal part (paragraph 13). So a listing with a seller's name, or a review with a username, brings that part into scope.
For the jobs that do collect people, here's how the draft's asks translate into a pipeline:
| The draft asks for | In your pipeline |
|---|---|
| Collection criteria | URL patterns per job |
| Syntax-based filtering | Regex drop at parse |
| Timestamped records | fetched_at on each row |
| A list of sources | Domains in config |
| Respect for opt-outs | robots.txt state logged |
Two of those rows deserve more than a table cell.
Filter at parse time, not in a cleanup job weeks later. Paragraph 38 names regular expressions for formats like social security and phone numbers, applied during collection. Data you never stored is data you never have to explain, and paragraph 60 is blunt about why that matters for AI: once a model is trained, personal data "cannot be easily deleted" from it.
Keep the source URL and the fetch time on every record. Paragraph 42 asks for timestamps for accuracy, but the same two fields answer every later question about where a fact came from. Training-data rules were already tightening under the AI Act (we wrote about that in April), and they keep asking that same question.
If you build for clients, you have to put the instructions in writing: sources, fields, exclusions. The draft treats documented instructions as what makes you a processor rather than a co-controller, and a one-page spec beats an argument about roles later.
What Happens After 30 October
The final text will change, and the bigger shift may come from Brussels rather than from the Board. On 21 October 2026 the Council agreed its negotiating position on the Digital Omnibus, the package that would amend the GDPR. According to IAPP's report on the mandate, member states added "clarifications on the lawful ground for processing personal data and need to maintain appropriate safeguards" for AI training data. Negotiations with the European Parliament come next. Whatever wording comes out, member states want the safeguards kept, which points to a balancing test much like this draft's.
We think the targeted and untargeted split survives both processes, because it's the one part of the draft that's about engineering rather than law. A regulator won't read your crawler's code. It can ask for your collection criteria, your source list and your timestamps, and a team that can hand those over the same afternoon has very little left to argue about.
So, the next question a privacy regulator asks a scraper won't be "how it works", but rather - "what it was told to collect".