# Smart Fetch (Auto)

> URL: https://foura.ai/docs/guides/smart-fetch
> Updated: 2026-11-13
> Index: https://foura.ai/llms.txt

You hand FourA a URL and a `validate` rule for what the real page should contain. FourA does the rest: it walks a cost-aware ladder, stops at the first rung that returns a response your rules accept, and remembers what worked per host so the next call on the same site is cheap.

This guide explains what auto does under the hood, when to use it, and how to read its response. For the parameter reference, see [API Endpoints](https://foura.ai/docs/api/endpoints#smart-fetch-auto).

## The Idea

Most scraping setups make you pick the engine up front. Single is fastest, Proxy adds rotation, Browser handles JavaScript. You guess wrong, you waste credits or you get blocked.

Auto flips it. You declare success (`validate`), not method. FourA climbs a ladder until one rung succeeds:

1. **Cheap probe** (single, straight out of FourA's own network)
2. **Browser**, straight out of FourA's own network, with JavaScript and a solver if the site challenges
3. **Rotated proxy single**
4. **Browser through proxy** for the hardest targets

Auto stops as soon as a rung returns a response your `validate` rule accepts.

One rung sits outside that order. When an exit reaches the site but the site refuses the deep URL you asked for, auto fetches the site's entry page through that same exit, keeps the cookies the entry page hands out, and asks for your URL again carrying them. That's the `warmup` rung. It only runs on a URL deeper than the site root, only after the direct attempt already failed, and it can only add an outcome, never take one away.

`forceProxy` defaults to `true`, so rungs 1 and 2 are skipped and the target never sees FourA's own address. Most calls then finish on rung 3, or on a replayed warm session. Set `forceProxy: false` when you know a target treats a clean address better than a rotating one, and rungs 1 and 2 come back.

## What You Send

The minimum is a URL plus a `validate` substring. Auto recognizes the common challenge pages on its own, but without `validate.data.accept` it can't tell a real page from a check page it doesn't know, or from a page that loaded without your content, and it may return either as success.

```bash
curl -X POST https://eu.api.foura.ai/api/auto/ \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}}
  }'
```

Optional knobs (see the [endpoint reference](https://foura.ai/docs/api/endpoints#smart-fetch-auto) for full details):

- `returnSession` (default `true`): return the winning `{ proxy, cookies, userAgent }` so you can replay it.
- `forceProxy` (default `true`): skip direct-egress rungs. Set `false` only if you know the site is friendlier to a clean IP than to rotating exits.
- `timeout_ms` (default `120000`): total budget for the whole call. The ladder portions it across rungs.
- `ignoreProxies`: proxy IDs to avoid on every sub-attempt.
- `followRedirects` (default `5`): max redirects on the cheap rungs.
- `headers`: headers to send to the site, as `[name, value]` pairs. They go out on the Single and Proxy requests, including the search for a working exit, and the browser rungs don't send them. Your own `Cookie` reaches the site unchanged, and a warm session's cookies are added beside it only under names you didn't send. When a warm session carries a `userAgent`, it replaces your `User-Agent`. Conditional headers such as `If-None-Match` and `If-Modified-Since` are left out of the exit search unless your `validate.status.accept` lists `304`.
- `exitClass`: `premium` lets Auto escalate to a premium exit once the standard pool can't reach the page; `standard` says it never escalates. Needs a plan that includes premium exits. See [exitClass on Smart Fetch](https://foura.ai/docs/api/endpoints#exitclass-on-smart-fetch).

## What You Get Back

```json
{
  "status": 200,
  "data": "<!doctype html>...",
  "headers": [{"content-type": "text/html"}],
  "meta": {
    "rung": "cache",
    "solved": false,
    "attempts": 1,
    "credits": 2,
    "attemptsByEngine": {"single": 1, "proxy": 0, "browser": 0},
    "browserRequests": 0
  },
  "session": {
    "proxy": "A1B2C3",
    "cookies": [{"name": "session", "value": "abc", "domain": "example.com"}],
    "userAgent": "Mozilla/5.0..."
  }
}
```

Three things to read:

- **`status` and `data`**: the target's answer. `data` is text whichever rung served the page, so a JSON page arrives as JSON text and you parse it yourself. `status` is the target's HTTP status, not the transport status of your call to FourA. For single and proxy rungs, `headers` is a per-hop array. For browser rungs, `headers` is a flat object.
- **`meta`**: the trace of what the ladder did, present on every response once the ladder has started. `meta.rung` names the step that delivered the response, `meta.attempts` counts sub-call tries, `meta.solved` flags whether a challenge page was completed, and `meta.credits` is the total spend for the call (the same number as the `X-FourA-Credits` header). `meta.attemptsByEngine` and `meta.browserRequests` say what the call sent: see [What a Call Spent](#what-a-call-spent). When the request named an `exitClass`, `meta.exitClass` says which class of exit delivered.
- **`session`**: the `{ proxy, cookies, userAgent }` triple that cracked the target. Use it to replay against the same host via `/api/single/` or `/api/browser/`. A field can be missing: a session won by a plain proxy fetch has `proxy` but no `userAgent`, and its `cookies` list holds only the cookies a site's check page set before the site served the page, so it's often empty. An answer from the cheap direct probe carries a `session` only when such a check page set cookies, and then it holds `cookies` alone. A status your `validate.status.accept` lists that an exit brought back (such as a removed page's `404`) has no `session` at all.

Auto answers with HTTP 200 whenever the ladder ran, even when every rung failed. Read `status` and `error` in the body to find out what happened, not the transport status code. A non-200 from `/api/auto/` means the call never reached the ladder: 401 for a bad key, 400 for a body that isn't valid JSON, a target on a private network, a target on FourA's own API host, or a URL with an unencoded space inside it, 403 with `X-FourA-Limit: plan_limit_premium` for `exitClass: premium` on a plan without premium exits, and 502, 503 or 504 when the service couldn't take the call or ran out of time. Auto takes no slot at the gateway, so the platform's shared limits don't refuse the call itself: when one refuses a call the ladder made, the answer is HTTP 200 with `status: 429` or `503` and `retryAfter` in the body. A field that fails validation also comes back as HTTP 200, with `status: 400`. A plan limit met inside the ladder also comes back as HTTP 200, with the refusal in the body (see [When Your Plan's Limits Meet the Ladder](#when-your-plans-limits-meet-the-ladder)).

## Replaying with the Session

After auto returns a session, you can drop straight into Single or Browser for follow-up pages on the same host. No new ladder climb, no new probe.

```python
import requests

API = "https://eu.api.foura.ai"
KEY = "YOUR_API_KEY"
H = {"X-API-Key": KEY, "Content-Type": "application/json"}

# 1) First call: let auto figure it out.
r = requests.post(f"{API}/api/auto/", headers=H, json={
    "url": "https://example.com/product/42",
    "validate": {"data": {"accept": ["Add to cart"]}},
}).json()

session = r.get("session") or {}
proxy = session.get("proxy")
user_agent = session.get("userAgent")

# 2) Follow-up pages: replay through single with the same proxy, UA and cookies.
for sku in ("43", "44", "45"):
    body = {"method": "GET", "url": f"https://example.com/product/{sku}"}
    if proxy:
        body["proxy"] = proxy
    headers = []
    if user_agent:
        headers.append(["User-Agent", user_agent])
    if session.get("cookies"):
        headers.append(["Cookie", "; ".join(f"{c['name']}={c['value']}" for c in session["cookies"])])
    if headers:
        body["headers"] = headers
    r = requests.post(f"{API}/api/single/", headers=H, json=body).json()
    print(sku, r["status"])
```

The session is only as durable as the target makes it. Some sites bind clearance to the cookie jar for hours; others rotate every few minutes. If a replay starts returning challenges again, call `/api/auto/` once more to refresh.

## When to Use Auto

| Use auto | Use single, proxy, or browser by hand |
|----------|----------------------------------------|
| You're targeting a new site and don't know what it needs | You already know the engine that works |
| You want one call that handles proxy and browser fallback for you | You want full control over per-call retries and timeouts |
| You're fine paying a few seconds of probing on the first call | First-call latency matters more than discovery |
| You want a learned session you can replay cheaply | You're optimizing a tight loop on a known-good target |

Auto isn't always the cheapest pick. If you know a target works with `single` + `unblocker`, calling Single directly is 2 credits with predictable latency. Auto on the same target costs whatever its ladder spends, which can be more if the site requires escalation.

## Validate Tells Auto What "Success" Means

The single most important parameter is `validate`. Without it, auto only rejects the challenge pages it recognizes, so an unfamiliar check page or an empty shell served with HTTP 200 passes as content.

Use `validate.data.accept` with a substring only the real page contains:

```json
{
  "validate": {
    "data": {
      "accept": ["sku-42-add-to-cart", "Customer reviews"]
    }
  }
}
```

For JSON APIs, accept a field name you expect:

```json
{
  "validate": {
    "data": { "accept": ["\"products\":["] },
    "status": { "accept": [200] }
  }
}
```

For sites that legitimately return non-200 (a country restriction you want to ignore, a removed page's `404` or `410`), allow them via `validate.status.accept`:

```json
{
  "validate": {
    "status": { "accept": [200, 451] }
  }
}
```

Auto returns a status you list in `validate.status.accept` as soon as an exit brings it back and it passes the rest of your rule. Four statuses are the exception: `202`, `403`, `429` and `503`. Auto reads those as the site checking who is asking, so an exit that brings one back doesn't end the search, and the ladder keeps climbing.

Without `validate`, auto falls back to "HTTP 200 = success" for every page it doesn't recognize as a challenge, so it won't catch an unfamiliar check page a site returns with a 200.

## Reading `meta.rung` to Understand What Happened

`meta.rung` is the most useful debug signal. Values:

- `probe` - solved on a cheap direct request. The cheapest path.
- `proxy` - needed proxy rotation to get through.
- `browser` - needed a full browser render, possibly with a challenge solve.
- `cache` - replayed a warm session from a prior auto call. Cheapest path on repeat calls.
- `warmup` - the site served its entry page but gated the deep URL, so auto fetched the entry page first, kept the cookies it handed out, and asked again with them. The session it stores from this rung isn't tied to one exit, so follow-up calls land on the cheap rungs.
- `fail` - no rung produced a response your rules accepted.

`meta.solved: true` means a challenge page was met and completed during the call. `meta.attempts` is the count of sub-call tries before success. For the detail behind it, read the `defense` field the single and proxy rungs return: see [Site checks](https://foura.ai/docs/api/site-checks).

If a site keeps ending on `browser` when you expected `probe`, consider whether a stricter `validate` rule (or a less strict one) would let a cheaper rung pass. Remember that `forceProxy` defaults to `true`, so the direct-egress probe is skipped unless you turn it off.

## What a Call Spent

`meta.attemptsByEngine` counts the requests the call sent to each engine, the same requests the Auto attempts panel shows on your [Overview](https://foura.ai/docs/dashboard/overview). `meta.browserRequests` counts the ones that took a slot of your daily browser allowance.

```json
"meta": {
  "rung": "browser",
  "solved": true,
  "attempts": 7,
  "credits": 47,
  "attemptsByEngine": {"single": 1, "proxy": 4, "browser": 3},
  "browserRequests": 2
}
```

| Field | Type | Description |
|-------|------|-------------|
| `meta.attemptsByEngine` | object | Requests sent to `single`, `proxy` and `browser`. A browser request your plan refused counts here, because it was sent. |
| `meta.browserRequests` | number | Browser requests that took a slot of your daily allowance. A browser request your plan refused takes nothing. |

`meta.attempts` counts the steps of the ladder, so it doesn't always equal the sum of `meta.attemptsByEngine`. One step can send a Single request and then a Proxy request, and a step whose browser is already shut for the day sends nothing.

Your running count for the day travels in the `X-FourA-Browser-Used-Today` and `X-FourA-Browser-Limit-Per-Day` headers: see [Response Headers](https://foura.ai/docs/api/response-headers).

## Errors and Edge Cases

When auto fails, the response carries `status` (usually the last failed rung's status) and an `error` string:

```json
{
  "status": 502,
  "error": "could not find a working exit for the target",
  "attempts": 7,
  "attemptReport": {
    "total": 7,
    "noResponse": 5,
    "defense": 2,
    "contentRejected": 0,
    "statusRejected": 0,
    "other": 0,
    "vendors": ["datadome"],
    "summary": "7 attempt(s): 5 got no answer from the site; 2 refused by the site (datadome)"
  },
  "meta": {
    "rung": "fail",
    "solved": false,
    "attempts": 7,
    "credits": 47,
    "attemptsByEngine": {"single": 0, "proxy": 4, "browser": 3},
    "browserRequests": 3
  }
}
```

`status` is the site's answer on the last attempt auto rejected, such as a `403`. When no attempt got an answer from the site at all, it's usually `502` or `504`, and `error` says whether no working exit was found or the `timeout_ms` budget ran out. `status: 0` only means the target's host name didn't resolve, and that answer has no `meta` because the ladder never started.

Auto also stops early when its browser keeps meeting a check it can't complete. After two exits refuse the browser that way and a search for an exit whose plain fetch passes your `validate` rule comes back empty, or after three such exits, the call ends. `status` holds the site's last answer and `error` starts with `the browser was refused by`, followed by the name of the check system.

Auto also stops when the site itself says the page isn't there. When two exits bring back the same `404` or `410` and your rules don't accept it, the call ends with that `status`, and `error` starts with `the site answered`. If a removed page is an answer you want, list `404` or `410` in `validate.status.accept`: Auto then returns it as delivered as soon as an exit brings it back.

After four Auto calls in a row with your API key end without the page because the site kept refusing, Auto stops trying that site for your key for 120 seconds. A call to it in that time answers at once with `status: 503`, `error: "target temporarily unservable, retry later"` and `retryAfter` in seconds, and costs nothing. It's counted per API key: other keys calling the same site aren't affected, and any call that gets the page resets the count. A call with `exitClass: premium` isn't stopped by it.

Check `attemptReport`, `meta.attemptsByEngine` and `meta.credits` to see where the budget went. If `meta.attempts` is high and `meta.rung` is `fail` after the browser rung, the target may need a longer `timeout_ms`, a stricter `validate` rule, or simply isn't reachable through rotating proxies right now.

### Why an Auto Call Failed

Every failed answer from a ladder that ran carries an `attemptReport`. It counts the engine requests that came back without the page, each one once, under what it met.

| Field | Type | Description |
|-------|------|-------------|
| `total` | integer | Requests that came back without the page. At most the sum of `meta.attemptsByEngine`, because a browser request your plan refused never tried the target. |
| `noResponse` | integer | Got no answer from the site |
| `defense` | integer | The site refused the request: a bot check was recognised, the page was a challenge, or the site answered 401, 403, 429 or 503 |
| `contentRejected` | integer | The page was fetched and only your `validate` rule rejected it |
| `statusRejected` | integer | The site answered with a status your request doesn't accept, such as 404 for a removed page |
| `other` | integer | Failed for another reason, such as a server error |
| `vendors` | string[] | Bot-check vendors recognised on the refusals the call met |
| `summary` | string | One sentence built from the counts, safe to log. It names the HTTP status the site answered where no vendor was recognised, and says when your daily browser allowance was spent, so no browser could be opened. |

The fields match the report on a failed [Proxy call](https://foura.ai/docs/troubleshooting/proxy-attempt-report), without `profilesTried`. Two things differ. Proxy counts each exit it tried, up to the `maxTries` you set, and Auto counts its own requests. Auto also reads the page and its status, so a refusal counts under `defense` even when no vendor was recognised. A count under `contentRejected` is the one you can fix yourself: the page came back and your rule didn't match it.

### When Your Plan's Limits Meet the Ladder

Auto's sub-calls are ordinary Single, Proxy and Browser requests under your key, so your [plan limits](https://foura.ai/docs/api/rate-limits) apply to them. The ladder reads the `X-FourA-Limit` code on a refusal and treats the two kinds differently.

**A shut rung leaves the rest of the ladder usable.** `plan_limit_browser_daily` (your Browser requests for the day are used up) and `plan_limit_concurrency` (that endpoint already has as many of your requests running as the plan allows) close one rung. Auto keeps working the other rungs, so you still get a page whenever a rotating exit or a warm session serves the content, and the exits it tried aren't blamed for a refusal that came from your own plan. Nothing is banned and no session is thrown away.

**An account that's out stops the ladder.** `plan_limit_credits`, `plan_limit_bandwidth`, `plan_limit_rate`, `plan_limit_feature`, and `plan_limit_premium` can't be helped by another rung, so auto returns at once instead of spending more of your credits proving it. The refusal comes back in the body with the sub-call's status and the same `reason` field the direct endpoints use:

```json
{
  "status": 429,
  "error": "Credit limit reached: 300,000 of 300,000 credits used this billing period. Upgrade your plan or wait for the reset.",
  "reason": "plan_limit_credits",
  "documentation": "https://foura.ai/prices",
  "used": 300000,
  "hard_stop": 300000,
  "retry_after_seconds": 86400,
  "resets_at": "2026-10-01T00:00:00.000Z",
  "meta": { "rung": "fail", "solved": false, "attempts": 1, "credits": 0 }
}
```

The whole refusal body from the sub-call comes through, plus `status`, `attemptReport` and `meta`. Read `status` from the body and not the transport status: auto still answers HTTP 200 here, because the ladder ran. A `plan_limit_feature` or `plan_limit_premium` refusal arrives the same way with `status: 403`. A refused sub-call spends nothing, so `meta.credits` counts only the rungs that got as far as the target.

One auto call can hold several slots while its ladder climbs, so a parallel batch of auto calls reaches a concurrency ceiling with fewer calls than you'd expect. [Run Requests in Parallel](https://foura.ai/docs/how-to/run-requests-in-parallel) covers sizing the batch.

## What Auto Does Not Do

- It does not change legal restrictions. If a site refuses every exit FourA can reach, auto returns that refusal.
- It does not cache content. Every call still hits the target. The "warm session" is the proxy and cookies, not the response.
- It is one row in the [Activity Log](https://foura.ai/docs/dashboard/activity-log), under the request id you received, with the sum of its sub-calls' credits. Open it and the Single / Proxy / Browser sub-calls auto made on your behalf are listed as its attempts, each with its own outcome. They count against your Single, Proxy and Browser limits, never against your request count or success rate.

## Related

- [API Endpoints](https://foura.ai/docs/api/endpoints#smart-fetch-auto): Full parameter reference
- [Choosing the Right Endpoint](https://foura.ai/docs/guides/choosing-task-type): When to pick auto vs single, proxy, or browser
- [Request Outcomes](https://foura.ai/docs/api/outcomes): Which outcomes are billable
- [Protected sites](https://foura.ai/docs/guides/protected-sites): What FourA does on sites that check who is asking
- [Site checks](https://foura.ai/docs/api/site-checks): The `defense` field behind `meta.solved`
- [MCP Recipes](https://foura.ai/docs/mcp/recipes): The same patterns as MCP tool calls
- [Rate Limits](https://foura.ai/docs/api/rate-limits): The plan limits auto's sub-calls are measured against
