← All posts
Use Case 5 min read

The Recrawl Problem: Keeping RAG Pipelines Fresh

Your RAG knowledge base ages out the week you ship it. Here's how teams recrawl hundreds of vertical sources without breaking their engineering budget.

ragai-trainingdata-collectionvertical-aiuse-case

The Challenge

Vertical AI startups all hit the same wall around month two. They ship a support copilot, a legal-research assistant, or a compliance bot. The first demo wins customers. Then the data ages out, and the answers start drifting from reality.

We've watched teams build the AI side cleanly and the data side as an afterthought. The ingestion pipeline is one Python script running on someone's laptop. It scrapes 200 source URLs once, dumps clean Markdown into a vector store, and everyone celebrates. Six weeks later, half the answers cite removed pages, deprecated APIs, or product features that shipped in March and shipped again in May.

The fix sounds simple: recrawl every source weekly. Reality is uglier. By 2026, around 60% of reputable sites block AI crawlers (up from 23% in late 2023), and the protections aren't dumb User-Agent checks anymore. They look at session behavior, request rhythm, and handshake-level signals. A naive script that worked in January is silently returning empty pages in March.

Worse, some sites now serve tarpit content (Markov-generated gibberish that reads like real prose) until it poisons your embeddings. So your engineers spend half their week patching the scraper instead of shipping product. Retrieval quality dips, customers notice, and the team you hired to build AI becomes a scraper maintenance shop.

The Approach

The recrawl problem splits into three concrete decisions that have to happen on every request:

  1. Render or not? Most documentation portals serve clean HTML. A growing share (anything built on Next.js, anything with client-side rendering) needs full browser rendering to return useful content.
  2. Which proxy? Residential, datacenter, mobile, geo-pinned, ISP-specific. The right pick changes by target.
  3. Did it actually work? A 200 with an empty body, or a CAPTCHA HTML page, is a successful HTTP request and a failed crawl.

A platform like FourA handles each of these as a first-class concern.

For the render decision, you call Single for the cheap, fast case and Browser for JS-heavy targets. The body of the call is the same shape, so your ingestion code branches once on a per-source flag instead of carrying a hundred site-specific quirks.

For proxy selection, Proxy Finder runs as part of every Single, Browser, and Auto call. The platform picks a working exit per request, returns its opaque id in the response (at r.proxy top-level on Single/Browser, or r.session.proxy on Auto), and you reuse that id on follow-up calls when you need to stick to the same exit. Your crawler doesn't carry a proxy ranking algorithm of its own. (We wrote about why pool size has stopped being the differentiator in Why Proxy Pool Size Stopped Mattering in 2026.)

And for the "did it actually work" question, every request supports a validate block. You declare what counts as success: accepted status codes, required header values, body strings that must or must not appear. FourA returns one of seven outcomes, and only success is billable. A 200 that fails your content rules is stamped application_fail and never enters your dataset.

Here's what a recrawl call looks like for a docs portal that needs a JS render. We let Auto orchestrate — it picks the right product (Single, Proxy, or Browser), handles bot defenses, and returns the session triple so the next recrawl can stick to the same exit:

import requests

r = requests.post(
    "https://api.foura.ai/api/auto",
    headers={"Authorization": "Bearer pk_live_..."},
    json={
        "url": "https://docs.example.com/changelog",
        "validate": {
            "status": {"accept": [200]},
            "data":   {"accept": ["<article"], "fail": ["captcha", "Just a moment"]},
        },
    },
).json()

# r["data"] or r["body"]   — rendered content (Auto runs the right sub-product per host;
#                            Single populates "data", Browser populates "body")
# r["session"]              — { "proxy": "<base36 id>", "cookies": [...], "userAgent": "..." }
# On the next recrawl, pass r["session"]["proxy"] back as `ignoreProxies: [<id>]` to avoid
# the same exit, or via /api/single with `proxy: <id>` to stick to it.

If the target throws a Cloudflare interstitial, the validate.data.fail rule catches it. The outcome stamped against your usage is application_fail. You don't pay for it, and your ingestion code knows to retry with a different proxy instead of feeding a "Just a moment..." page into the embeddings.

For the wider corpus, you wrap the same pattern in your existing job queue. Teams we've talked to run nightly diffs against the previous crawl, re-embed only the documents that actually changed, and refresh 500-source corpora in a couple of hours of wall-clock time. The job queue stays yours. The proxy churn, the render decision, the success verdict are ours.

Results

What the freshness loop looks like once the infrastructure stops being the bottleneck (illustrative scenario based on patterns we see across vertical AI teams):

  • 500 source URLs recrawled weekly, instead of a 200-URL one-shot at launch
  • Engineering time on the scraper: under 2 hours per week, down from 1-2 days
  • Retrieval staleness window: 5-7 days, instead of unbounded
  • Garbage rate in the vector store near zero, because Cloudflare interstitials and tarpit pages get rejected at the validate layer before they reach your embedding model
  • Cost predictable per source, because failed crawls don't show up in billing

The point isn't that any of these are magic. The point is that they're boring. And boring is what production AI needs. (For more on where the math stops working with hosted LLM extraction, see When LLM Extraction Stops Paying for Itself.)

Key Takeaway

Most teams building vertical AI think the moat is the prompt, the model choice, or the retrieval algorithm. It isn't. The moat is the freshness loop: the unglamorous infrastructure that keeps the knowledge base honest week after week.

The teams that win in vertical AI through 2026 won't be the ones with the cleverest prompts. They'll be the ones whose users never notice the data is current, because it always is.