Docs › Guides › Handling Anti-Bot Protection

Handling Anti-Bot Protection

Modern websites use advanced bot detection. This guide explains how FourA handles anti-bot systems and how to maximize your success rate.

How Bot Detection Works

Websites use several layers of protection:

  • IP reputation: Data centers and known proxy addresses get blocked
  • Wire fingerprinting: Each HTTP client has a unique handshake signature that sites can detect
  • Browser fingerprinting: JavaScript checks for headless browser indicators
  • Behavioral analysis: Request patterns, timing, and navigation flow
  • CAPTCHAs: Visual challenges as a last line of defense

Common anti-bot providers include Cloudflare, DataDome, PerimeterX, and Akamai Bot Manager.

Fastest Path: Auto

If you don't know the protection level yet, call /api/auto/ with a validate.data.accept substring that only the real page carries. Auto walks a cost-aware ladder (cheap probe, rotated proxy, browser render, browser through proxy) and stops at the first rung that returns a response your rules accept. On repeat calls to the same host, a warm session is replayed instead, so the second hit is cheap.

curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://protected-site.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'

Without validate.data.accept, auto can't tell a real 200 from a challenge interstitial returned with status 200 and may return the challenge as success. See the Smart Fetch guide for the full walkthrough.

How FourA Helps at Each Layer

Realistic Wire-Level Requests

The single endpoint (POST /api/single/) emits handshake characteristics that match a real browser. This passes the most common wire-level checks without the overhead of running a full browser.

Enable unblocker to also inject realistic browser headers (User-Agent, Sec-Ch-Ua, Sec-Fetch-*, Accept-Encoding). unblocker is on by default; set false only to send a plain client signature.

{
  "method": "GET",
  "url": "https://protected-site.com/data",
  "unblocker": true
}

Real Browser Rendering

The browser endpoint (POST /api/browser/) runs a full Chrome browser instance. Combined with anti-detection patches, it passes most JavaScript-based fingerprint checks. unblocker on Browser triggers the auto defense solver (Turnstile clearance and similar gates); leave it on unless you want the raw challenge page back.

Proxy Rotation

The proxy endpoint (POST /api/proxy/) automatically rotates through residential and data center proxies. If one IP gets blocked, the next attempt uses a different one. Use ignoreProxies on a follow-up call to skip exits you already burned; use maxTries (default 5, max 90) to control how hard it works.

Country-Scoped Exits

Pass exitCountries on /api/proxy/ to restrict selection to proxies whose target-visible country matches a strict allowlist. Values are two-letter codes (["CZ", "GB"]), trimmed, uppercased, and deduplicated. FourA never falls back to an unrequested country; if the current pool has no match, the response returns code: "no_eligible_proxy" with the normalized scope in details.exitCountries so you can retry later without loosening the requirement.

curl -X POST https://eu.api.foura.ai/api/proxy/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 5,
    "exitCountries": ["CZ", "GB"],
    "request": {"method": "GET", "url": "https://target.example/pricing"}
  }'

The response includes exitCountry when scoping was requested. Verify it belongs to your allowlist before trusting the payload, then reuse the returned proxy ID on any follow-up Browser call so the JavaScript render happens through the same exit.

FourA Tells You What Stopped You

You don't have to guess which system blocked a request. When a target runs a bot check, the response names it.

  • POST /api/single/ and POST /api/proxy/ return a defense object: defense.vendor is the system, defense.solved says whether the check was cleared, and defense.present lists everything recognised on that response.
  • POST /api/browser/ returns defenseSolved plus defenses.present and defenses.cleared.
{
  "status": 200,
  "data": "<!doctype html>...",
  "defense": {
    "vendor": "sgcaptcha",
    "solved": true,
    "present": ["sgcaptcha"],
    "cookie": "_I_=<clearance>"
  }
}

Two rules follow from it:

  1. solved: false means the body may be the challenge, not the page. FourA never dresses an unsolved challenge up as content, so check the flag before parsing.
  2. A solve hands you the clearance. When defense.cookie is present, send it back as a Cookie header on the same exit with the same User-Agent and the follow-up requests skip the check entirely.

FourA recognises Cloudflare, SiteGround, DataDome, PerimeterX, Akamai, Incapsula, AWS WAF, hCaptcha, reCAPTCHA, and eBay's own challenge. Recognising is wider than clearing: a system we can name but not clear is reported and never changes what the request costs. See Anti-Bot Defenses for every field, the current clear-versus-detect split, and a replay example.

Strategy by Protection Level

Unknown Protection

Use auto. It probes cheap first and only escalates as far as the target forces it, so you pay for the discovery once per host.

Low Protection (most sites)

Use the single endpoint with unblocker. The wire-level match is enough.

curl -X POST https://eu.api.foura.ai/api/single/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"method": "GET", "url": "https://news-site.com/article", "unblocker": true}'

Medium Protection (Cloudflare, basic WAF)

Use the browser endpoint to pass JavaScript challenges:

curl -X POST https://eu.api.foura.ai/api/browser/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://protected-site.com/data", "timeout_ms": 15000}'

High Protection (DataDome, PerimeterX)

Use the proxy endpoint with multiple retry attempts:

curl -X POST https://eu.api.foura.ai/api/proxy/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 10,
    "request": {
      "method": "GET",
      "url": "https://heavily-protected.com/prices",
      "unblocker": true
    }
  }'

For chained WAF challenges (Cloudflare "Just a moment", Vercel Security Checkpoint, Akamai Bot Manager) where you need the rendered page after the challenge clears, see the MCP Recipes. The "Protected page: proxy first, browser when JavaScript is needed" recipe shows the exact proxy-then-browser handoff.

Best Practices

  1. Start with auto for unknown targets. Pass a validate rule, let the ladder pick the cheapest rung, then read meta.rung in the response to see which engine worked. Once you know, call that engine directly for repeat traffic.

  2. Reuse the winning session. After an auto call, the returned session (proxy ID + cookies + userAgent) can be replayed through /api/single/ or /api/browser/ for follow-up pages on the same host, at Single's price.

  3. Respect rate limits. Even with proxy rotation, sending hundreds of requests per second to a single site will trigger behavioral detection. Space your requests by at least 1 to 2 seconds.

  4. Keep unblocker on. For Single, Proxy, and Browser requests, unblocker: true (the default) injects realistic browser headers automatically. Turn it off only when you specifically need a plain client signature or a raw challenge page.

  5. Monitor success rates. Check the Dashboard metrics to track your success rate over time. A sudden drop usually means the target site updated its protection.

  6. Skip burned exits. If a /api/proxy/ or /api/auto/ call returned a proxy ID that then started failing, pass it in ignoreProxies on the next call so FourA picks a different exit.

  7. Read defense before you retry. The vendor name tells you whether a different browser profile is worth trying, whether you need a full render, or whether the check is one nobody clears without a solving service.

  8. Change the browser you present. Some targets accept one browser and refuse another. Set browser, os, or version on Single and Proxy, and read GET /api/profiles for the current catalogue. Details in the endpoint reference.

What FourA Can't Bypass

Some scenarios require additional handling outside the API:

  • Login-protected content: FourA doesn't manage long-lived logins for you. The browser endpoint accepts cookies per-request; carry your session cookies in yourself.
  • Interactive CAPTCHAs: reCAPTCHA and hCaptcha need a solving service. FourA recognises both and reports them in defense.present, but doesn't clear them. Turnstile is handled by Browser.
  • Geo-restricted content: use exitCountries on /api/proxy/ to pin selection to allowed countries. Sites that additionally restrict by ISP or ASN (some country-licensed bookmakers, certain government services) may still block generic residential exits; the request returns no_eligible_proxy when the current pool has no matching exit.
  • Sites with legal restrictions: Always ensure your data collection complies with the target site's terms of service and applicable laws.

Next Steps

Last updated: August 12, 2026
← Choosing the Right Endpoint Smart Fetch (Auto) →