← All posts
Use Case 5 min read

SERP Monitoring at Scale After num=100

Tracking Google rankings at scale got harder once num=100 died. Here's how SEO engineering teams are rebuilding SERP monitoring infrastructure for 2026.

serp-monitoringrank-trackingseoweb-scrapinggoogle-search

The Challenge

If your team builds rank trackers, SEO dashboards, or competitive intelligence tools, 2026 broke your unit economics. Google quietly retired the num=100 URL parameter on Google Search this year, the trick every SERP scraper used to pull 100 results in one request. Now the same coverage takes ten requests instead of one.

That's the obvious cost. The hidden costs are nastier.

Rank tracking only works when you see the SERP a real searcher would see in the right country, region, and city. A keyword that ranks #4 in London might rank #11 in Edinburgh and #19 in Belfast. Local 3-packs, shopping carousels, news boxes, knowledge panels, AI Overviews. Every SERP feature shifts with geography and device. (Scrape.do measured AI Overview text appearing in roughly 36% of queries during early 2026.) If your scraper routes through a proxy in the wrong city, your ranking data is fiction told confidently.

So a defensible SERP product in 2026 needs three things working together: a request that looks like a real browser at the wire level, a proxy that lives in the exact city you're trying to monitor, and the ability to render JavaScript when Google decides to load half the result client-side. Miss any of the three and your data degrades quietly.

The FourA Approach

The bottleneck in SERP scraping at scale isn't the request. It's the routing.

Most homegrown pipelines start with a fixed proxy pool and treat the query as the variable. With Google's geographic targeting, it's the other way around. The query is what you have. The proxy is what you have to get right.

We've watched teams pattern this on top of FourA roughly like this:

  1. Proxy Finder keeps a working pool of proxies validated against fresh liveness checks and tagged with country, region, city, and ASN. When a request needs to come from Manchester or Boston or São Paulo, Proxy Finder picks one that actually lives there and is alive on the last check. The picking happens before the fetch, not during it. For more on why that routing layer matters, see our piece on Smart Proxy Routing.

  2. Single handles the SERP fetch itself. For standard organic results, raw HTML is plenty. Set unblocker: true and the request gets through handshake-level anti-bot walls without you needing to know which signature Google is checking that week. We broke down what the flag does on the wire in our Web Unblocker post.

  3. Browser handles SERPs where critical content shows up after JavaScript runs. AI Overviews, expanded shopping packs, knowledge-panel content, sticky local 3-packs. Same URL, same target, the request just runs through a full browser session and returns the fully painted page. (Plus screenshots, which save you the day an SEO lead asks why your dashboard says #3 but they see #6 in their browser.)

A single call against the proxy-routed API:

curl -X POST "https://api.foura.ai/api/proxy" \
  -H "x-api-key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 3,
    "timeout_ms": 20000,
    "request": {
      "method": "GET",
      "url": "https://www.google.co.uk/search?q=plumber+manchester&hl=en-GB",
      "unblocker": true,
      "validate": {
        "status": { "accept": [200] },
        "data": { "fail": ["unusual traffic", "/sorry/", "captcha"] }
      }
    }
  }'

That's three concerns separated cleanly: geo-correct proxy work (Proxy Finder), the request itself (Single), JavaScript rendering when you need it (Browser). Your code doesn't carry proxy-health logic or guess which IP is still alive at 3am. That's somebody else's problem.

And store every response keyed by (keyword, location, device, timestamp). That's the actual unit of truth for rank tracking. Not "we ranked here for this keyword today," but "we ranked here for this keyword, from this city, on this device, at this minute." Without that level of attribution, two days of data can quietly contradict each other and you'll have no way to tell which one was right. SEO teams that monitor protected verticals are already living with this. We've also written about how bot detection went behavioral, which adds a fourth axis (session continuity) for sites that look at request sequencing rather than per-request signals.

Results

A rank tracker monitoring 5,000 keywords across 12 cities, twice daily, was about 120,000 requests per day under the old num=100 regime. Now it's closer to 1.2 million, by simple pagination math (illustrative scenario based on industry benchmarks).

Teams that ported this shape to a three-product stack tend to report:

  • 40-60% reduction in per-request cost versus running their own proxy pool, mostly because they stopped paying for proxy churn, dead IPs, and the engineering hours to maintain the rotation.
  • City-level location accuracy moving from ~70% to over 95%, because Proxy Finder filters by city and verifies liveness on the last check before handing the proxy over.
  • No special path for AI Overviews. A keyword that fetches via Single can be promoted to Browser without rewriting the pipeline. The contract is identical: URL in, response out.

You don't need any of this for ten keywords and a laptop. But you need it once the pipeline is monitoring tens of thousands of keywords across countries, your customers refresh the dashboard at 9am Monday, and the rankings have to be real.

Key Takeaway

The hard part of SERP monitoring stopped being the request a long time ago. It's been the routing. Whose city are you fetching from? Is that IP alive? Did Google return the layout a real searcher in that location would actually see, or the empty one they serve when they smell a scraper?

If you're an SEO team running rank tracking on a stack you built yourself, the question for 2026 isn't whether to scrape Google. You already do. The question is whether your infrastructure can keep producing trustworthy rankings when the rules change without warning, and how much of your engineering team you're willing to spend keeping it that way.