# Cloudflare Verified Bots: Graded by Bot, Revoked by Company

> URL: https://foura.ai/blog/cloudflare-verified-bots-trust-bands
> Updated: 2026-11-11
> Index: https://foura.ai/llms.txt

## One Bot's Record Can Cost the Whole Company

Cloudflare explained its new bot grading with a driver's license: it [tells you who someone is](https://blog.cloudflare.com/verified-bots-accountability/), not whether they'll drive responsibly next year. Good analogy. It's missing one detail, though. A suspended license takes one driver off the road, and under Cloudflare's rules a bad record can take every bot a company runs.

On October 16, Cloudflare said it now evaluates Verified bots continuously, on two signals so far: whether their traffic stays consistent with a site's robots.txt over time, and whether they slow down after an HTTP 429. The results become a "trust band". Engineers reading that will reach for the retry logic first, and [Crawlbase's long piece on a fetch loop that scores well](https://crawlbase.com/blog/the-web-has-started-grading-crawlers/) is a good place to start.

We think retry logic is the smaller half. If you build scrapers for other people, the question that matters is whose name is on the listing.

### What Actually Shipped

Less than the headlines suggest. Cloudflare says it's "already using this system internally" while it builds history. Public scores on Radar and a threshold site owners can choose are both written in the future tense, with no date attached. Nothing Cloudflare has published lets a site turn your crawler away because of a trust band yet.

The scope is narrow too. Only bots on the Verified list are in it, and everything else isn't scored at all. That's nearly everything: at Cloudflare Connect in October, the company said [fewer than one in ten automated requests](https://www.shashi.co/2026/10/cloudflare-warns-ai-agents-will-make.html) comes from a bot that identifies itself.

So if your scraper isn't on that list, this announcement doesn't grade you.

**What this means:** the ledger isn't open to site owners yet, and most automated traffic isn't in it. The people who should care now are the ones with a listed crawler, or a plan to get one.

### The Penalty Has a Wider Radius Than the Score

Here's the part the write-ups we read skip. The trust band describes one bot. The consequence doesn't have to stop there. Cloudflare's [Verified bots documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), updated November 2, says it "may revoke Verified status for all bots and agents owned or operated by an organization" after repeated violations, and it names the case of a company that "complies with the policy for one bot or agent while violating it with another."

Read that as a small agency. One company, several client projects, one name in the directory. A client wants a catalog by Friday and asks you to turn the rate up. If that job keeps its pace through the 429s it draws, and keeps doing it, the cost needn't stay on that job. It can land on the company, and on every other project running under it.

The same page shows why this suddenly applies to commercial work. BotBase, Cloudflare's directory, now lists a "Data Collection" behavior, defined as "price scraping, competitive intelligence gathering, and third-party analytics." It sits beside Search and Training. Commercial collection can be a declared, named activity, and we count that as good news.

But look at the labels next to it. A bot is either Direct (one operator, usually on its own infrastructure) or Intermediary ("an agentic service that a wide range of end users can operate"). Cloudflare calls the problem with the second group transitive trust: you might trust the operator without trusting everyone driving it, so it's experimenting with forwarding details about the end user. A scraping agency is an intermediary in everything except the paperwork.

### A 429 Isn't Always About Speed

The second signal sounds simple. Cloudflare calls a 429 "a directive for the visitor to reduce its request rate" and watches whether traffic slows or pauses afterward.

But sites use that status for more than load. [Crawl Readiness tested the Tranco top 1,000](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026) on October 12, asking each homepage for a page as a browser and then as four AI crawlers, about ten requests a site with nothing retried. Of the 78 sites that refused a crawler their own robots.txt allowed, 14 answered with a 429 and nothing else (one pass, user-agent strings only, so treat it as a hint).

The authors set those aside as possible rate limits, which is the careful thing to do. We read them differently. A site that hands a browser a 200 and a named crawler a 429, after a handful of requests, probably isn't reporting congestion. It's stating a preference in the vocabulary of congestion.

So a grader that treats every 429 as one instruction will sometimes measure the wrong thing, and the site rarely tells you which case you're in. A concurrency cap, a per-minute limit and a flat no all arrive as the same three digits.

The common client doesn't help. Plenty of retry code waits out `Retry-After` and then carries on at the rate that drew the refusal. Crawlbase's local test puts a number on it: against a limit of two requests a second, a loop that held its old rate drew 13 refusals in 40 requests, while one that lowered its ceiling drew 3 and still fetched more pages per second (1.11 against 0.94). It's a toy server. The direction holds anyway.

**What this means:** if your bot's on the Verified list, slowing down is the answer the score rewards, and it's cheaper than it looks. But log what you got, because some of those 429s were never about your speed.

### Signing and Obeying Are Coming Apart

Google's documentation makes the tension visible. Its [Web Bot Auth page](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), last updated October 13, says a subset of Google-Agent's requests are signed, as an experiment. Its page on [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), where Google-Agent is listed, says that because a user requested the fetch, these fetchers "generally ignore robots.txt rules."

So Google is proving who sent a request while declining the file everyone else may be graded on. Cloudflare's post doesn't say how a user-directed fetch will be judged. If you're building agents, that boundary is the one to watch, because "a person asked for this" is about to carry a lot of weight.

## What This Means for People Who Build Scrapers for Clients

**Sort your work by whose name is on it.** A product with its own public crawler and one clear purpose is a good candidate for Verified status. A fleet running other people's jobs under your company's name is a harder call, because one client's job can put the listing at risk for all of them.

**Put crawl conduct in the contract.** If a client can ask for a higher rate, the contract should say who carries the consequence. Once a 429 becomes a record, "just go faster" stops being a free request.

**Keep your own log.** Status, `Retry-After`, the rate you held before and after, per host. Cloudflare promises an explainable band, but it hasn't said how an operator disputes one, and without your own numbers the grader's account is the only account. It's the same habit [the EDPB's draft guidelines](https://foura.ai/blog/edpb-web-scraping-guidelines-targeted) reward: being able to show what you collected, from where, and when.

**Stop reading robots.txt as a permission slip.** Cloudflare's post says it directly: "there is no implication that anything not stated is allowed." [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) was never a license, and now someone's keeping score against it.

## The Dropdown Nobody Opens

In August we wrote that [the web was sorting bots by purpose, not behavior](https://foura.ai/blog/search-agent-training-bot-rules), and that defaults are how a policy becomes the norm. Behavior arrived two months later. And the default effect is already measurable: Crawlbase's October 30 sweep of the Tranco top 2,000 found 71 sites with a Content-Signal line in robots.txt, 39 of them carrying the identical Cloudflare managed default. [Pay-per-crawl](https://foura.ai/blog/pay-per-crawl-splits-the-web) needed the same thing this does: a buyer with a name.

Our bet: when trust thresholds ship, most sites will take whatever value comes preselected, and a site's opinion of your crawler will mostly be a dropdown nobody opened. We're on both ends of this status code ourselves. When a plan limit stops a call, the FourA API [names the limit](https://foura.ai/docs/api/rate-limits) and sends `Retry-After` only where waiting helps, and Proxy Finder rests a rate-limited exit for that one site, for the wait its `Retry-After` names. A 429 was always a request. It's about to become a record.
