Smart Fetch (Auto)

You hand FourA a URL and a validate rule for what the real page should contain. FourA does the rest: it walks a cost-aware ladder, stops at the first rung that returns a response your rules accept, and remembers what worked per host so the next call on the same site is cheap.

This guide explains what auto does under the hood, when to use it, and how to read its response. For the parameter reference, see API Endpoints.

The Idea

Most scraping setups make you pick the engine up front. Single is fastest, Proxy adds rotation, Browser handles JavaScript. You guess wrong, you waste credits or you get blocked.

Auto flips it. You declare success (validate), not method. FourA climbs a ladder until one rung succeeds:

  1. Cheap probe (single, straight out of FourA's own network)
  2. Browser, straight out of FourA's own network, with JavaScript and a solver if the site challenges
  3. Rotated proxy single
  4. Browser through proxy for the hardest targets

Auto stops as soon as a rung returns a response your validate rule accepts.

One rung sits outside that order. When an exit reaches the site but the site refuses the deep URL you asked for, auto fetches the site's entry page through that same exit, keeps the cookies the entry page hands out, and asks for your URL again carrying them. That's the warmup rung. It only runs on a URL deeper than the site root, only after the direct attempt already failed, and it can only add an outcome, never take one away.

forceProxy defaults to true, so rungs 1 and 2 are skipped and the target never sees FourA's own address. Most calls then finish on rung 3, or on a replayed warm session. Set forceProxy: false when you know a target treats a clean address better than a rotating one, and rungs 1 and 2 come back.

What You Send

The minimum is a URL plus a validate substring. Auto recognizes the common challenge pages on its own, but without validate.data.accept it can't tell a real page from a check page it doesn't know, or from a page that loaded without your content, and it may return either as success.

curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'

Optional knobs (see the endpoint reference for full details):

  • returnSession (default true): return the winning { proxy, cookies, userAgent } so you can replay it.
  • forceProxy (default true): skip direct-egress rungs. Set false only if you know the site is friendlier to a clean IP than to rotating exits.
  • timeout_ms (default 120000): total budget for the whole call. The ladder portions it across rungs.
  • ignoreProxies: proxy IDs to avoid on every sub-attempt.
  • followRedirects (default 5): max redirects on the cheap rungs.
  • headers: headers to send to the site, as [name, value] pairs. They go out on the Single and Proxy requests, including the search for a working exit, and the browser rungs don't send them. Your own Cookie reaches the site unchanged, and a warm session's cookies are added beside it only under names you didn't send. When a warm session carries a userAgent, it replaces your User-Agent. Conditional headers such as If-None-Match and If-Modified-Since are left out of the exit search unless your validate.status.accept lists 304.
  • exitClass: premium lets Auto escalate to a premium exit once the standard pool can't reach the page; standard says it never escalates. Needs a plan that includes premium exits. See exitClass on Smart Fetch.

What You Get Back

{
  "status": 200,
  "data": "<!doctype html>...",
  "headers": [{"content-type": "text/html"}],
  "meta": {
    "rung": "cache",
    "solved": false,
    "attempts": 1,
    "credits": 2,
    "attemptsByEngine": {"single": 1, "proxy": 0, "browser": 0},
    "browserRequests": 0
  },
  "session": {
    "proxy": "A1B2C3",
    "cookies": [{"name": "session", "value": "abc", "domain": "example.com"}],
    "userAgent": "Mozilla/5.0..."
  }
}

Three things to read:

  • status and data: the target's answer. data is text whichever rung served the page, so a JSON page arrives as JSON text and you parse it yourself. status is the target's HTTP status, not the transport status of your call to FourA. For single and proxy rungs, headers is a per-hop array. For browser rungs, headers is a flat object.
  • meta: the trace of what the ladder did, present on every response once the ladder has started. meta.rung names the step that delivered the response, meta.attempts counts sub-call tries, meta.solved flags whether a challenge page was completed, and meta.credits is the total spend for the call (the same number as the X-FourA-Credits header). meta.attemptsByEngine and meta.browserRequests say what the call sent: see What a Call Spent. When the request named an exitClass, meta.exitClass says which class of exit delivered.
  • session: the { proxy, cookies, userAgent } triple that cracked the target. Use it to replay against the same host via /api/single/ or /api/browser/. A field can be missing: a session won by a plain proxy fetch has proxy but no userAgent, and its cookies list holds only the cookies a site's check page set before the site served the page, so it's often empty. An answer from the cheap direct probe carries a session only when such a check page set cookies, and then it holds cookies alone. A status your validate.status.accept lists that an exit brought back (such as a removed page's 404) has no session at all.

Auto answers with HTTP 200 whenever the ladder ran, even when every rung failed. Read status and error in the body to find out what happened, not the transport status code. A non-200 from /api/auto/ means the call never reached the ladder: 401 for a bad key, 400 for a body that isn't valid JSON, a target on a private network, a target on FourA's own API host, or a URL with an unencoded space inside it, 403 with X-FourA-Limit: plan_limit_premium for exitClass: premium on a plan without premium exits, and 502, 503 or 504 when the service couldn't take the call or ran out of time. Auto takes no slot at the gateway, so the platform's shared limits don't refuse the call itself: when one refuses a call the ladder made, the answer is HTTP 200 with status: 429 or 503 and retryAfter in the body. A field that fails validation also comes back as HTTP 200, with status: 400. A plan limit met inside the ladder also comes back as HTTP 200, with the refusal in the body (see When Your Plan's Limits Meet the Ladder).

Replaying with the Session

After auto returns a session, you can drop straight into Single or Browser for follow-up pages on the same host. No new ladder climb, no new probe.

import requests

API = "https://eu.api.foura.ai"
KEY = "YOUR_API_KEY"
H = {"X-API-Key": KEY, "Content-Type": "application/json"}

# 1) First call: let auto figure it out.
r = requests.post(f"{API}/api/auto/", headers=H, json={
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}},
}).json()

session = r.get("session") or {}
proxy = session.get("proxy")
user_agent = session.get("userAgent")

# 2) Follow-up pages: replay through single with the same proxy, UA and cookies.
for sku in ("43", "44", "45"):
    body = {"method": "GET", "url": f"https://example.com/product/{sku}"}
    if proxy:
        body["proxy"] = proxy
    headers = []
    if user_agent:
        headers.append(["User-Agent", user_agent])
    if session.get("cookies"):
        headers.append(["Cookie", "; ".join(f"{c['name']}={c['value']}" for c in session["cookies"])])
    if headers:
        body["headers"] = headers
    r = requests.post(f"{API}/api/single/", headers=H, json=body).json()
    print(sku, r["status"])

The session is only as durable as the target makes it. Some sites bind clearance to the cookie jar for hours; others rotate every few minutes. If a replay starts returning challenges again, call /api/auto/ once more to refresh.

When to Use Auto

Use auto Use single, proxy, or browser by hand
You're targeting a new site and don't know what it needs You already know the engine that works
You want one call that handles proxy and browser fallback for you You want full control over per-call retries and timeouts
You're fine paying a few seconds of probing on the first call First-call latency matters more than discovery
You want a learned session you can replay cheaply You're optimizing a tight loop on a known-good target

Auto isn't always the cheapest pick. If you know a target works with single + unblocker, calling Single directly is 2 credits with predictable latency. Auto on the same target costs whatever its ladder spends, which can be more if the site requires escalation.

Validate Tells Auto What "Success" Means

The single most important parameter is validate. Without it, auto only rejects the challenge pages it recognizes, so an unfamiliar check page or an empty shell served with HTTP 200 passes as content.

Use validate.data.accept with a substring only the real page contains:

{
  "validate": {
    "data": {
      "accept": ["sku-42-add-to-cart", "Customer reviews"]
    }
  }
}

For JSON APIs, accept a field name you expect:

{
  "validate": {
    "data": { "accept": ["\"products\":["] },
    "status": { "accept": [200] }
  }
}

For sites that legitimately return non-200 (a country restriction you want to ignore, a removed page's 404 or 410), allow them via validate.status.accept:

{
  "validate": {
    "status": { "accept": [200, 451] }
  }
}

Auto returns a status you list in validate.status.accept as soon as an exit brings it back and it passes the rest of your rule. Four statuses are the exception: 202, 403, 429 and 503. Auto reads those as the site checking who is asking, so an exit that brings one back doesn't end the search, and the ladder keeps climbing.

Without validate, auto falls back to "HTTP 200 = success" for every page it doesn't recognize as a challenge, so it won't catch an unfamiliar check page a site returns with a 200.

Reading meta.rung to Understand What Happened

meta.rung is the most useful debug signal. Values:

  • probe - solved on a cheap direct request. The cheapest path.
  • proxy - needed proxy rotation to get through.
  • browser - needed a full browser render, possibly with a challenge solve.
  • cache - replayed a warm session from a prior auto call. Cheapest path on repeat calls.
  • warmup - the site served its entry page but gated the deep URL, so auto fetched the entry page first, kept the cookies it handed out, and asked again with them. The session it stores from this rung isn't tied to one exit, so follow-up calls land on the cheap rungs.
  • fail - no rung produced a response your rules accepted.

meta.solved: true means a challenge page was met and completed during the call. meta.attempts is the count of sub-call tries before success. For the detail behind it, read the defense field the single and proxy rungs return: see Site checks.

If a site keeps ending on browser when you expected probe, consider whether a stricter validate rule (or a less strict one) would let a cheaper rung pass. Remember that forceProxy defaults to true, so the direct-egress probe is skipped unless you turn it off.

What a Call Spent

meta.attemptsByEngine counts the requests the call sent to each engine, the same requests the Auto attempts panel shows on your Overview. meta.browserRequests counts the ones that took a slot of your daily browser allowance.

"meta": {
  "rung": "browser",
  "solved": true,
  "attempts": 7,
  "credits": 47,
  "attemptsByEngine": {"single": 1, "proxy": 4, "browser": 3},
  "browserRequests": 2
}
Field Type Description
meta.attemptsByEngine object Requests sent to single, proxy and browser. A browser request your plan refused counts here, because it was sent.
meta.browserRequests number Browser requests that took a slot of your daily allowance. A browser request your plan refused takes nothing.

meta.attempts counts the steps of the ladder, so it doesn't always equal the sum of meta.attemptsByEngine. One step can send a Single request and then a Proxy request, and a step whose browser is already shut for the day sends nothing.

Your running count for the day travels in the X-FourA-Browser-Used-Today and X-FourA-Browser-Limit-Per-Day headers: see Response Headers.

Errors and Edge Cases

When auto fails, the response carries status (usually the last failed rung's status) and an error string:

{
  "status": 502,
  "error": "could not find a working exit for the target",
  "attempts": 7,
  "attemptReport": {
    "total": 7,
    "noResponse": 5,
    "defense": 2,
    "contentRejected": 0,
    "statusRejected": 0,
    "other": 0,
    "vendors": ["datadome"],
    "summary": "7 attempt(s): 5 got no answer from the site; 2 refused by the site (datadome)"
  },
  "meta": {
    "rung": "fail",
    "solved": false,
    "attempts": 7,
    "credits": 47,
    "attemptsByEngine": {"single": 0, "proxy": 4, "browser": 3},
    "browserRequests": 3
  }
}

status is the site's answer on the last attempt auto rejected, such as a 403. When no attempt got an answer from the site at all, it's usually 502 or 504, and error says whether no working exit was found or the timeout_ms budget ran out. status: 0 only means the target's host name didn't resolve, and that answer has no meta because the ladder never started.

Auto also stops early when its browser keeps meeting a check it can't complete. After two exits refuse the browser that way and a search for an exit whose plain fetch passes your validate rule comes back empty, or after three such exits, the call ends. status holds the site's last answer and error starts with the browser was refused by, followed by the name of the check system.

Auto also stops when the site itself says the page isn't there. When two exits bring back the same 404 or 410 and your rules don't accept it, the call ends with that status, and error starts with the site answered. If a removed page is an answer you want, list 404 or 410 in validate.status.accept: Auto then returns it as delivered as soon as an exit brings it back.

After four Auto calls in a row with your API key end without the page because the site kept refusing, Auto stops trying that site for your key for 120 seconds. A call to it in that time answers at once with status: 503, error: "target temporarily unservable, retry later" and retryAfter in seconds, and costs nothing. It's counted per API key: other keys calling the same site aren't affected, and any call that gets the page resets the count. A call with exitClass: premium isn't stopped by it.

Check attemptReport, meta.attemptsByEngine and meta.credits to see where the budget went. If meta.attempts is high and meta.rung is fail after the browser rung, the target may need a longer timeout_ms, a stricter validate rule, or simply isn't reachable through rotating proxies right now.

Why an Auto Call Failed

Every failed answer from a ladder that ran carries an attemptReport. It counts the engine requests that came back without the page, each one once, under what it met.

Field Type Description
total integer Requests that came back without the page. At most the sum of meta.attemptsByEngine, because a browser request your plan refused never tried the target.
noResponse integer Got no answer from the site
defense integer The site refused the request: a bot check was recognised, the page was a challenge, or the site answered 401, 403, 429 or 503
contentRejected integer The page was fetched and only your validate rule rejected it
statusRejected integer The site answered with a status your request doesn't accept, such as 404 for a removed page
other integer Failed for another reason, such as a server error
vendors string[] Bot-check vendors recognised on the refusals the call met
summary string One sentence built from the counts, safe to log. It names the HTTP status the site answered where no vendor was recognised, and says when your daily browser allowance was spent, so no browser could be opened.

The fields match the report on a failed Proxy call, without profilesTried. Two things differ. Proxy counts each exit it tried, up to the maxTries you set, and Auto counts its own requests. Auto also reads the page and its status, so a refusal counts under defense even when no vendor was recognised. A count under contentRejected is the one you can fix yourself: the page came back and your rule didn't match it.

When Your Plan's Limits Meet the Ladder

Auto's sub-calls are ordinary Single, Proxy and Browser requests under your key, so your plan limits apply to them. The ladder reads the X-FourA-Limit code on a refusal and treats the two kinds differently.

A shut rung leaves the rest of the ladder usable. plan_limit_browser_daily (your Browser requests for the day are used up) and plan_limit_concurrency (that endpoint already has as many of your requests running as the plan allows) close one rung. Auto keeps working the other rungs, so you still get a page whenever a rotating exit or a warm session serves the content, and the exits it tried aren't blamed for a refusal that came from your own plan. Nothing is banned and no session is thrown away.

An account that's out stops the ladder. plan_limit_credits, plan_limit_bandwidth, plan_limit_rate, plan_limit_feature, and plan_limit_premium can't be helped by another rung, so auto returns at once instead of spending more of your credits proving it. The refusal comes back in the body with the sub-call's status and the same reason field the direct endpoints use:

{
  "status": 429,
  "error": "Credit limit reached: 300,000 of 300,000 credits used this billing period. Upgrade your plan or wait for the reset.",
  "reason": "plan_limit_credits",
  "documentation": "https://foura.ai/prices",
  "used": 300000,
  "hard_stop": 300000,
  "retry_after_seconds": 86400,
  "resets_at": "2026-10-01T00:00:00.000Z",
  "meta": { "rung": "fail", "solved": false, "attempts": 1, "credits": 0 }
}

The whole refusal body from the sub-call comes through, plus status, attemptReport and meta. Read status from the body and not the transport status: auto still answers HTTP 200 here, because the ladder ran. A plan_limit_feature or plan_limit_premium refusal arrives the same way with status: 403. A refused sub-call spends nothing, so meta.credits counts only the rungs that got as far as the target.

One auto call can hold several slots while its ladder climbs, so a parallel batch of auto calls reaches a concurrency ceiling with fewer calls than you'd expect. Run Requests in Parallel covers sizing the batch.

What Auto Does Not Do

  • It does not change legal restrictions. If a site refuses every exit FourA can reach, auto returns that refusal.
  • It does not cache content. Every call still hits the target. The "warm session" is the proxy and cookies, not the response.
  • It is one row in the Activity Log, under the request id you received, with the sum of its sub-calls' credits. Open it and the Single / Proxy / Browser sub-calls auto made on your behalf are listed as its attempts, each with its own outcome. They count against your Single, Proxy and Browser limits, never against your request count or success rate.
Last updated: November 13, 2026