Docs › Guides › Handling Anti-Bot Protection

Handling Anti-Bot Protection

Modern websites use advanced bot detection. This guide explains how FourA handles anti-bot systems and how to maximize your success rate.

How Bot Detection Works

Websites use several layers of protection:

  • IP reputation: Data centers and known proxy IPs get blocked
  • Wire fingerprinting: Each HTTP client has a unique handshake signature that sites can detect
  • Browser fingerprinting: JavaScript checks for headless browser indicators
  • Behavioral analysis: Request patterns, timing, and navigation flow
  • CAPTCHAs: Visual challenges as a last line of defense

Common anti-bot providers include Cloudflare, DataDome, PerimeterX, and Akamai Bot Manager.

Fastest Path: Auto

If you don't know the protection level yet, call /api/auto/ with a validate.data.accept substring that only the real page carries. Auto walks a cost-aware ladder (cheap probe, rotated proxy, browser render, browser through proxy) and stops at the first rung that returns a response your rules accept. On repeat calls to the same host, a warm session is replayed instead, so the second hit is cheap.

curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://protected-site.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'

Without validate.data.accept, auto can't tell a real 200 from a challenge interstitial returned with status 200 and may return the challenge as success. See the Smart Fetch guide for the full walkthrough.

How FourA Helps at Each Layer

Realistic Wire-Level Requests

The single endpoint (POST /api/single/) emits handshake characteristics that match a real browser. This passes the most common wire-level checks without the overhead of running a full browser.

Enable unblocker to also inject realistic browser headers (User-Agent, Sec-Ch-Ua, Sec-Fetch-*, Accept-Encoding). unblocker is on by default; set false only to send a plain client signature.

{
  "method": "GET",
  "url": "https://protected-site.com/data",
  "unblocker": true
}

Real Browser Rendering

The browser endpoint (POST /api/browser/) runs a full Chrome browser instance. Combined with anti-detection patches, it passes most JavaScript-based fingerprint checks. unblocker on Browser triggers the auto defense solver (Turnstile clearance and similar gates); leave it on unless you want the raw challenge page back.

Proxy Rotation

The proxy endpoint (POST /api/proxy/) automatically rotates through residential and data center proxies. If one IP gets blocked, the next attempt uses a different one. Use ignoreProxies on a follow-up call to skip exits you already burned; use maxTries (default 5, max 90) to control how hard it works.

Strategy by Protection Level

Unknown Protection

Use auto. It probes cheap first and only escalates as far as the target forces it, so you pay for the discovery once per host.

Low Protection (most sites)

Use the single endpoint with unblocker. The wire-level match is enough.

curl -X POST https://eu.api.foura.ai/api/single/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"method": "GET", "url": "https://news-site.com/article", "unblocker": true}'

Medium Protection (Cloudflare, basic WAF)

Use the browser endpoint to pass JavaScript challenges:

curl -X POST https://eu.api.foura.ai/api/browser/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://protected-site.com/data", "timeout_ms": 15000}'

High Protection (DataDome, PerimeterX)

Use the proxy endpoint with multiple retry attempts:

curl -X POST https://eu.api.foura.ai/api/proxy/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 10,
    "request": {
      "method": "GET",
      "url": "https://heavily-protected.com/prices",
      "unblocker": true
    }
  }'

For tier-1 WAF chains (Vercel Security Checkpoint, Cloudflare "Just a moment", Akamai Bot Manager) where you need the rendered page after the challenge clears, see the WAF challenge recipe for the chained proxy → browser pattern.

Best Practices

  1. Start with auto for unknown targets. Pass a validate rule, let the ladder pick the cheapest rung, then read meta.rung in the response to see which engine worked. Once you know, call that engine directly for repeat traffic.

  2. Reuse the winning session. After an auto call, the returned session (proxy ID + cookies + userAgent) can be replayed through /api/single/ or /api/browser/ for follow-up pages on the same host, at Single's price.

  3. Respect rate limits. Even with proxy rotation, sending hundreds of requests per second to a single site will trigger behavioral detection. Space your requests by at least 1 to 2 seconds.

  4. Keep unblocker on. For single and proxy requests, unblocker: true (the default) injects realistic browser headers automatically. Turn it off only when you specifically need a plain client signature.

  5. Monitor success rates. Check the Dashboard metrics to track your success rate over time. A sudden drop usually means the target site updated its protection.

  6. Skip burned exits. If a /api/proxy/ or /api/auto/ call returned a proxy ID that then started failing, pass it in ignoreProxies on the next call so FourA picks a different exit.

What FourA Can't Bypass

Some scenarios require additional handling outside the API:

  • Login-protected content: FourA doesn't manage long-lived logins for you. The browser endpoint accepts cookies per-request; carry your session cookies in yourself.
  • Interactive CAPTCHAs: reCAPTCHA v3 and hCaptcha require solving services (Turnstile is handled).
  • Country / ASN allowlists: country-licensed sites (online bookmakers, some government services) only accept traffic from a small set of ISPs. Proxy rotation won't help; you'll need a different data source.
  • Sites with legal restrictions: Always ensure your data collection complies with the target site's terms of service and applicable laws.

Next Steps

Last updated: July 8, 2026
← Choosing the Right Endpoint Smart Fetch (Auto) →