Docs › Guides › Protected sites

Protected sites

Modern websites use advanced bot detection. This guide explains how FourA gets you the page on sites that check who is asking, and how to raise your success rate.

How Bot Detection Works

Websites use several layers of protection:

  • IP reputation: Data centers and known proxy addresses get blocked
  • Wire fingerprinting: Each HTTP client has a unique handshake signature that sites can detect
  • Browser fingerprinting: JavaScript checks for headless browser indicators
  • Behavioral analysis: Request patterns, timing, and navigation flow
  • Verification pages: a visual task the visitor has to complete

The response names the system that ran a check; Site checks lists them.

Fastest Path: Auto

If you don't know the protection level yet, call /api/auto/ with a validate.data.accept substring that only the real page carries. Auto walks a cost-aware ladder (a rotated proxy, then a browser through a proxy; with forceProxy: false, a cheap direct probe and a direct browser render come first) and stops at the first rung that returns a response your rules accept. On repeat calls to the same host, a warm session is replayed instead, so the second hit is cheap.

curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://protected-site.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'

Auto recognizes the common challenge pages and keeps climbing when it meets one. Your validate.data.accept string catches the rest: a check page it doesn't know, a login wall, or a page without the content you need. See the Smart Fetch guide for the full walkthrough.

How FourA Helps at Each Layer

Realistic Wire-Level Requests

The single endpoint (POST /api/single/) emits handshake characteristics that match a real browser. Sites answer it the way they answer a browser, without the overhead of running one.

Enable unblocker to also inject realistic browser headers (User-Agent, Sec-Ch-Ua, Sec-Fetch-*, Accept-Encoding). unblocker is on by default; set false only to send a plain client signature.

{
  "method": "GET",
  "url": "https://protected-site.com/data",
  "unblocker": true
}

Real Browser Rendering

The browser endpoint (POST /api/browser/) runs a full Chrome browser instance. It runs the page's JavaScript the way a visitor's browser does. unblocker on Browser completes the checks a page asks for before it loads (Turnstile and similar gates); leave it on unless you want the challenge page back as it came.

Proxy Rotation

The proxy endpoint (POST /api/proxy/) automatically rotates through residential and data center proxies. If one IP gets blocked, the next attempt uses a different one. Use ignoreProxies on a follow-up call to skip exits you already burned; use maxTries (default 5, max 90) to control how hard it works.

Country-Scoped Exits

Pass exitCountries on /api/proxy/ to restrict selection to proxies whose target-visible country matches a strict allowlist. Values are two-letter codes (["CZ", "GB"]), trimmed, uppercased, and deduplicated. FourA never falls back to an unrequested country; if the current pool has no match, the response returns code: "no_eligible_proxy" with the normalized scope in details.exitCountries so you can retry later without loosening the requirement. Country scoping is included from the Startup plan up. On a plan without it, a call that sends exitCountries is refused with a 403 and X-FourA-Limit: plan_limit_feature.

curl -X POST https://eu.api.foura.ai/api/proxy/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 5,
    "exitCountries": ["CZ", "GB"],
    "request": {"method": "GET", "url": "https://target.example/pricing"}
  }'

The response includes exitCountry when scoping was requested. Verify it belongs to your allowlist before trusting the payload, then reuse the returned proxy ID on any follow-up Browser call so the JavaScript render happens through the same exit.

FourA Tells You What Stopped You

You don't have to guess which system blocked a request. When a target runs a bot check, the response names it.

  • POST /api/single/ and POST /api/proxy/ return a defense object: defense.vendor is the system, defense.solved says whether the check was cleared, and defense.present lists everything recognised on that response.
  • POST /api/browser/ returns defenseSolved plus defenses.present and defenses.cleared.
{
  "status": 200,
  "data": "<!doctype html>...",
  "defense": {
    "vendor": "sgcaptcha",
    "solved": true,
    "present": ["sgcaptcha"],
    "cookie": "_I_=<clearance>"
  }
}

Two rules follow from it:

  1. solved: false means the body may be the challenge, not the page. FourA never presents a challenge page as content, so check the flag before parsing.
  2. A solve hands you the clearance. When defense.cookie is present, send it back as a Cookie header on the same exit with the same User-Agent and the follow-up requests skip the check entirely.

FourA recognises the common check systems, including the first-party checks of eBay, Reddit, Amazon and Google Search. Recognising is wider than clearing: a system we can name but not clear is reported and never raises what the request costs. When the page that came back is that system's check page, even at HTTP 200, the request isn't billed and the X-FourA-Check-Page header names it. See Site checks for every field, the current clear-versus-detect split, and a replay example.

Strategy by Protection Level

Unknown Protection

Use auto. It probes cheap first and only escalates as far as the target forces it, so you pay for the discovery once per host.

Low Protection (most sites)

Use the single endpoint with unblocker. The wire-level match is enough.

curl -X POST https://eu.api.foura.ai/api/single/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"method": "GET", "url": "https://news-site.com/article", "unblocker": true}'

Medium: a challenge page or a basic firewall

Use the browser endpoint to pass JavaScript challenges:

curl -X POST https://eu.api.foura.ai/api/browser/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://protected-site.com/data", "timeout_ms": 15000}'

High: behaviour and fingerprint checks

Use the proxy endpoint with multiple retry attempts:

curl -X POST https://eu.api.foura.ai/api/proxy/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "maxTries": 10,
    "request": {
      "method": "GET",
      "url": "https://heavily-protected.com/prices",
      "unblocker": true
    }
  }'

For chained challenge pages ("Just a moment", a security checkpoint) where you need the rendered page after the challenge clears, see the MCP Recipes. The "Protected page: proxy first, browser when JavaScript is needed" recipe shows the exact proxy-then-browser handoff.

Best Practices

  1. Start with auto for unknown targets. Pass a validate rule, let the ladder pick the cheapest rung, then read meta.rung in the response to see which engine worked. Once you know, call that engine directly for repeat traffic.

  2. Reuse the winning session. After an auto call, the returned session (proxy ID + cookies + userAgent) can be replayed through /api/single/ or /api/browser/ for follow-up pages on the same host, at that endpoint's own price: 2 credits on Single with unblocker, 5 on Browser (10 for an interactive page).

  3. Respect rate limits. Even with proxy rotation, sending hundreds of requests per second to a single site will trigger behavioral detection. Space your requests by at least 1 to 2 seconds.

  4. Keep unblocker on. On Single and Proxy, unblocker: true (the default) sends a realistic browser signature and headers. On Browser it turns on the challenge solver. Turn it off only when you specifically need a plain client signature or a raw challenge page.

  5. Monitor success rates. Check the Dashboard metrics to track your success rate over time. A sudden drop usually means the target site updated its protection.

  6. Skip burned exits. If a /api/proxy/ or /api/auto/ call returned a proxy ID that then started failing, pass it in ignoreProxies on the next call so FourA picks a different exit.

  7. Read defense before you retry. The vendor name tells you whether a different browser profile is worth trying, whether you need a full render, or whether the check is one nobody clears without a solving service.

  8. Change the browser you present. Some targets accept one browser and refuse another. Set browser, os, or version on Single and Proxy, and read GET /api/profiles for the current catalogue. Details in the endpoint reference.

Limits

Some scenarios require additional handling outside the API:

  • Login-protected content: FourA doesn't manage long-lived logins for you. The browser endpoint accepts cookies per-request; carry your session cookies in yourself.
  • Interactive verification tasks: FourA recognises the visual ones and reports them in defense.present, but doesn't complete them. Turnstile is handled by Browser.
  • Content limited to certain countries: use exitCountries on /api/proxy/ to pin selection to allowed countries. Sites that additionally restrict by ISP or ASN (some country-licensed bookmakers, certain government services) may still block generic residential exits; the request returns no_eligible_proxy when the current pool has no matching exit.
  • Sites with legal restrictions: Always ensure your data collection complies with the target site's terms of service and applicable laws.

Next Steps

Last updated: September 27, 2026
← Choosing the Right Endpoint How Your Plan Is Metered →