Docs › Guides › Smart Fetch (Auto)

Smart Fetch (Auto)

You hand FourA a URL and a validate rule for what the real page should contain. FourA does the rest: it walks a cost-aware ladder, stops at the first rung that returns a response your rules accept, and remembers what worked per host so the next call on the same site is cheap.

This guide explains what auto does under the hood, when to use it, and how to read its response. For the parameter reference, see API Endpoints.

The Idea

Most scraping setups make you pick the engine up front. Single is fastest, Proxy adds rotation, Browser handles JavaScript. You guess wrong, you waste credits or you get blocked.

Auto flips it. You declare success (validate), not method. FourA climbs a ladder until one rung succeeds:

  1. Cheap probe (single, straight out of FourA's own network)
  2. Browser, straight out of FourA's own network, with JavaScript and a solver if the site challenges
  3. Rotated proxy single
  4. Browser through proxy for the hardest targets

Auto stops as soon as a rung returns a response your validate rule accepts.

One rung sits outside that order. When an exit reaches the site but the site refuses the deep URL you asked for, auto fetches the site's entry page through that same exit, keeps the cookies the entry page hands out, and asks for your URL again carrying them. That's the warmup rung. It only runs on a URL deeper than the site root, only after the direct attempt already failed, and it can only add an outcome, never take one away.

forceProxy defaults to true, so rungs 1 and 2 are skipped and the target never sees FourA's own address. Most calls then finish on rung 3, or on a replayed warm session. Set forceProxy: false when you know a target treats a clean address better than a rotating one, and rungs 1 and 2 come back.

What You Send

The minimum is a URL plus a validate substring. Auto recognizes the common challenge pages on its own, but without validate.data.accept it can't tell a real page from a check page it doesn't know, or from a page that loaded without your content, and it may return either as success.

curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'

Optional knobs (see the endpoint reference for full details):

  • returnSession (default true): return the winning { proxy, cookies, userAgent } so you can replay it.
  • forceProxy (default true): skip direct-egress rungs. Set false only if you know the site is friendlier to a clean IP than to free rotating proxies.
  • timeout_ms (default 120000): total budget for the whole call. The ladder portions it across rungs.
  • ignoreProxies: proxy IDs to avoid on every sub-attempt.
  • followRedirects (default 5): max redirects on the cheap rungs.

What You Get Back

{
  "status": 200,
  "data": "<!doctype html>...",
  "headers": [{"content-type": "text/html"}],
  "meta": {
    "rung": "cache",
    "solved": false,
    "attempts": 1,
    "credits": 2
  },
  "session": {
    "proxy": "A1B2C3",
    "cookies": [{"name": "session", "value": "abc", "domain": "example.com"}],
    "userAgent": "Mozilla/5.0..."
  }
}

Three things to read:

  • status and data: the target's answer. data is text on every rung: a JSON page comes back as a JSON string even when a browser served it, so parse it on your side. status is the target's HTTP status, not the transport status of your call to FourA. For single and proxy rungs, headers is a per-hop array. For browser rungs, headers is a flat object.
  • meta: the trace of what the ladder did, present on every response once the ladder has started. meta.rung names the step that delivered the response, meta.attempts counts sub-call tries, meta.solved flags whether a challenge page was completed, and meta.credits is the total spend for the call (the same number as the X-FourA-Credits header).
  • session: the { proxy, cookies, userAgent } triple that cracked the target. Use it to replay against the same host via /api/single/ or /api/browser/.

Auto answers with HTTP 200 whenever the ladder ran, even when every rung failed. Read status and error in the body to find out what happened, not the transport status code. A non-200 from /api/auto/ means the call never reached the ladder: 401 for a bad key, 400 for a body that isn't valid JSON or a target on a private network, and 502, 503 or 504 when the service couldn't take the call or ran out of time. Auto takes no slot at the gateway, so the platform's shared limits don't refuse the call itself: when one refuses a call the ladder made, the answer is HTTP 200 with status: 429 or 503 and retryAfter in the body. A field that fails validation also comes back as HTTP 200, with status: 400. A plan limit met inside the ladder also comes back as HTTP 200, with the refusal in the body (see When Your Plan's Limits Meet the Ladder).

Replaying with the Session

After auto returns a session, you can drop straight into Single or Browser for follow-up pages on the same host. No new ladder climb, no new probe.

import requests

API = "https://eu.api.foura.ai"
KEY = "YOUR_API_KEY"
H = {"X-API-Key": KEY, "Content-Type": "application/json"}

# 1) First call: let auto figure it out.
r = requests.post(f"{API}/api/auto/", headers=H, json={
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}},
}).json()

session = r["session"]
proxy = session["proxy"]
user_agent = session["userAgent"]

# 2) Follow-up pages: replay through single with the same proxy + UA.
for sku in ("43", "44", "45"):
    r = requests.post(f"{API}/api/single/", headers=H, json={
        "method": "GET",
        "url": f"https://example.com/product/{sku}",
        "proxy": proxy,
        "headers": [["User-Agent", user_agent]],
    }).json()
    print(sku, r["status"])

The session is only as durable as the target makes it. Some sites bind clearance to the cookie jar for hours; others rotate every few minutes. If a replay starts returning challenges again, call /api/auto/ once more to refresh.

When to Use Auto

Use auto Use single, proxy, or browser by hand
You're targeting a new site and don't know what it needs You already know the engine that works
You want one call that handles direct, proxy, and browser fallback for you You want full control over per-call retries and timeouts
You're fine paying a few seconds of probing on the first call First-call latency matters more than discovery
You want a learned session you can replay cheaply You're optimizing a tight loop on a known-good target

Auto isn't always the cheapest pick. If you know a target works with single + unblocker, calling Single directly is 2 credits with predictable latency. Auto on the same target costs whatever its ladder spends, which can be more if the site requires escalation.

Validate Tells Auto What "Success" Means

The single most important parameter is validate. Without it, auto only rejects the challenge pages it recognizes, so an unfamiliar check page or an empty shell served with HTTP 200 passes as content.

Use validate.data.accept with a substring only the real page contains:

{
  "validate": {
    "data": {
      "accept": ["sku-42-add-to-cart", "Customer reviews"]
    }
  }
}

For JSON APIs, accept a field name you expect:

{
  "validate": {
    "data": { "accept": ["\"products\":["] },
    "status": { "accept": [200] }
  }
}

For sites that legitimately return non-200 (a country restriction you want to ignore, an intentional 403 on logged-out endpoints), allow them via validate.status.accept:

{
  "validate": {
    "status": { "accept": [200, 451] }
  }
}

Without validate, auto falls back to "HTTP 200 = success" for every page it doesn't recognize as a challenge, so it won't catch an unfamiliar check page a site returns with a 200.

Reading meta.rung to Understand What Happened

meta.rung is the most useful debug signal. Values:

  • probe - solved on a cheap direct request. The cheapest path.
  • proxy - needed proxy rotation to get through.
  • browser - needed a full browser render, possibly with a challenge solve.
  • cache - replayed a warm session from a prior auto call. Cheapest path on repeat calls.
  • warmup - the site served its entry page but gated the deep URL, so auto fetched the entry page first, kept the cookies it handed out, and asked again with them. The session it stores from this rung isn't tied to one exit, so follow-up calls land on the cheap rungs.
  • fail - no rung produced a response your rules accepted.

meta.solved: true means a challenge page was met and completed during the call. meta.attempts is the count of sub-call tries before success. For the detail behind it, read the defense field the single and proxy rungs return: see Site checks.

If a site keeps ending on browser when you expected probe, consider whether a stricter validate rule (or a less strict one) would let a cheaper rung pass. Remember that forceProxy defaults to true, so the direct-egress probe is skipped unless you turn it off.

Errors and Edge Cases

When auto fails, the response carries status (usually the last failed rung's status) and an error string:

{
  "status": 502,
  "error": "could not find a working exit for the target",
  "attempts": 7,
  "meta": {
    "rung": "fail",
    "solved": false,
    "attempts": 7,
    "credits": 47
  }
}

status is the site's answer on the last attempt auto rejected, such as a 403. When no attempt got an answer from the site at all, it's usually 502 or 504, and error says whether no working exit was found or the timeout_ms budget ran out. status: 0 only means the target's host name didn't resolve, and that answer has no meta because the ladder never started.

Check meta.attempts and meta.credits to see where the budget went. If meta.attempts is high and meta.rung is fail after the browser rung, the target may need a longer timeout_ms, a stricter validate rule, or simply isn't reachable through rotating proxies right now.

When Your Plan's Limits Meet the Ladder

Auto's sub-calls are ordinary Single, Proxy and Browser requests under your key, so your plan limits apply to them. The ladder reads the X-FourA-Limit code on a refusal and treats the two kinds differently.

A shut rung leaves the rest of the ladder usable. plan_limit_browser_daily (your Browser requests for the day are used up) and plan_limit_concurrency (that endpoint already has as many of your requests running as the plan allows) close one rung. Auto keeps working the other rungs, so you still get a page whenever a rotating exit or a warm session serves the content, and the exits it tried aren't blamed for a refusal that came from your own plan. Nothing is banned and no session is thrown away.

An account that's out stops the ladder. plan_limit_credits, plan_limit_bandwidth, plan_limit_rate, plan_limit_feature, and plan_limit_premium can't be helped by another rung, so auto returns at once instead of spending more of your credits proving it. The refusal comes back in the body with the sub-call's status and the same reason field the direct endpoints use:

{
  "status": 429,
  "error": "Credit limit reached: 75,000 of 75,000 credits used this billing period. Upgrade your plan or wait for the reset.",
  "reason": "plan_limit_credits",
  "documentation": "https://foura.ai/prices",
  "used": 75000,
  "hard_stop": 75000,
  "retry_after_seconds": 86400,
  "resets_at": "2026-10-01T00:00:00.000Z",
  "meta": { "rung": "fail", "solved": false, "attempts": 1, "credits": 0 }
}

The whole refusal body from the sub-call comes through, plus status and meta. Read status from the body and not the transport status: auto still answers HTTP 200 here, because the ladder ran. A plan_limit_feature or plan_limit_premium refusal arrives the same way with status: 403. A refused sub-call spends nothing, so meta.credits counts only the rungs that got as far as the target.

One auto call can hold several slots while its ladder climbs, so a parallel batch of auto calls reaches a concurrency ceiling with fewer calls than you'd expect. Run Requests in Parallel covers sizing the batch.

What Auto Does Not Do

  • It does not change legal restrictions. If a site refuses every exit FourA can reach, auto returns that refusal.
  • It does not cache content. Every call still hits the target. The "warm session" is the proxy and cookies, not the response.
  • It is one row in the Activity Log, under the request id you received, with the sum of its sub-calls' credits. Open it and the Single / Proxy / Browser sub-calls auto made on your behalf are listed as its attempts, each with its own outcome. They count against your Single, Proxy and Browser limits, never against your request count or success rate.
Last updated: September 30, 2026
← How Your Plan Is Metered