← All posts
Product Update 5 min read

Redirect Control and Raw Buffer Mode

FourA's API now supports configurable redirect limits and raw binary responses. Two options that change how you handle real-world scraping edge cases.

apifeaturesredirectsbinary-data

Redirect chains break scrapers. Binary responses corrupt when decoded as text. Two problems that come up constantly once you're past the "fetch a page, parse the HTML" stage.

We shipped two new request options to handle both: followRedirects and returnBuffer. They're live on the API now.

How It Works

Redirect Control with followRedirects

Most scraping APIs handle redirects as a boolean: follow them or don't. That works until you hit a redirect chain that loops, or you need the intermediate 302 response itself to extract a tracking parameter.

FourA's followRedirects takes an integer between 0 and 20. Omit it (or set 0), and you get the raw redirect response back, headers and all. Set it to 5, and the request follows up to five hops before returning whatever it lands on.

curl -X POST "https://eu.api.foura.ai/v1/request" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/short-link",
    "followRedirects": 3,
    "chromeHeaders": true
  }'

This follows up to three redirects. If the chain resolves in two, you get the final page. If it's longer than three, you get whatever the third hop returned.

The distinction matters more than you'd think. E-commerce sites redirect through tracking URLs before landing on the product page. You want to follow those. But affiliate networks and URL shorteners sometimes create chains that go six, seven, eight hops deep. And some redirect loops never resolve at all. Capping at a specific number means you collect data without getting stuck in an infinite loop that burns through your request timeout.

Before this, the workaround was sending a request with redirects disabled, manually parsing the Location header, and sending another request. That's two API calls minimum, twice the latency, and code you have to maintain. Now it's one call with a number.

Raw Binary Responses with returnBuffer

When you're collecting images, PDFs, or protobuf payloads, text decoding destroys the data. The HTTP library assumes the response is text, applies charset detection, and silently mangles every byte that doesn't fit. Protobuf becomes unreadable. Image headers break. You end up with corrupted files and no obvious error message to explain why.

returnBuffer tells the API to skip text decoding entirely.

curl -X POST "https://eu.api.foura.ai/v1/request" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product-image.jpg",
    "returnBuffer": true
  }'

The response body comes back as raw bytes (base64-encoded in JSON responses). Decode it on your end and you have exactly what the server sent. No charset assumptions, no encoding conversion, no silent corruption.

This was one of the more common support tickets we saw: users collecting product images or PDF catalogs and getting files that wouldn't open. The fix was always the same, but now there's a flag for it instead of a workaround.

Impact

Both features cut the number of API calls per job. followRedirects eliminates manual redirect-chasing loops. returnBuffer eliminates the "fetch, realize it's corrupted, re-fetch with different settings" cycle.

For redirect-heavy targets (affiliate links, URL shorteners, e-commerce tracking chains), we've seen request counts drop by 40-60% in early testing when users switch from manual redirect handling to followRedirects. And for binary collection tasks (product images, document downloads), returnBuffer turns a multi-step workaround into a single option (early results).

These aren't flashy features. They're the kind of thing you don't think about until your scraper breaks at 3am because a site added an extra redirect hop to their checkout flow.

For Power Users

Combine followRedirects with response validation for precise control over redirect chains. Follow redirects, but fail the request if the final destination hits a wall:

curl -X POST "https://eu.api.foura.ai/v1/request" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/12345",
    "followRedirects": 5,
    "chromeHeaders": true,
    "validate": {
      "status": { "fail": [403, 503] },
      "data": { "fail": ["Access Denied", "captcha"] }
    }
  }'

This follows up to five redirects, then checks the final response. If the site redirected you to a CAPTCHA page or an access-denied wall, the request fails cleanly. No garbage data to filter out downstream.

For binary collection, pair returnBuffer with HEAD requests when you need to check content types before downloading large files. FourA handles HEAD correctly (preventing common libcurl errors internally), so you can inspect headers without fetching the body. Check the Content-Type, decide if it's worth downloading, then make the full request with returnBuffer: true.

And if you're using browser tasks for JavaScript-heavy targets, note that these options apply to the direct HTTP engine. Browser requests handle redirects through Chrome's built-in navigation, which follows them by default with no cap.

What's Next

We're working on exposing more request-level controls through the API: custom DNS resolution, per-phase timeout tuning, and certificate handling options. The goal is full curl-impersonate power through a clean REST interface, without the infrastructure overhead.

If there's a specific option you need, we're listening. The dashboard already shows how your requests perform with these new options, so you can measure the difference yourself.