← All posts
Use Case 6 min read

Tender Monitoring: The Notices Your Scraper Never Saw

Tender monitoring below the EU threshold means scraping a long tail of small portals. The costly failure isn't a refusal, it's a green run that stored nothing.

tender-monitoringpublic-procurementweb-scrapingdata-qualityuse-case

€216,000. Below that line, a European city council buying services doesn't have to publish its tender on TED, the EU's central register.

Which means the contracts a small bidder can realistically win live somewhere else: on the portal of whichever council, region, utility or ministry is doing the buying.

The Challenge

More than 250,000 public authorities in the EU spend around €2.6 trillion a year on works, supplies and services, by the European Commission's count. The EU directives only govern contracts above the thresholds, which from 1 January 2026 sit at €140,000 for central government supplies and services, €216,000 for local and regional authorities, and €5,404,000 for works. Below them, national rules apply and publishing on TED is voluntary.

That's why tender monitoring is a scraping job at all. If you're building it for a bid consultancy or a construction firm, the above-threshold part is the easy part. The rest is a long tail of portals nobody designed for a machine to read.

BidPrime, which sells public-sector bid data in North America, called the US version of this "thousands of workflows" in May 2026: portals that throttle automated access, procurement systems that need an account or a session, attachments stored apart from the notice, URLs that change without warning. Nothing on that list is specific to the US.

But the failure that hurts in tender monitoring isn't the portal that refuses you. That one shows up in your logs. It's the run that comes back green with nothing in it.

A council portal can answer an expired session with a friendly page at HTTP 200. A maintenance notice can render where the results table should be. A search you submitted yesterday can hand you page one again, because page two only existed for the cookie that asked for it. Your parser finds zero rows and writes "no new tenders today". And unlike a missed price, a missed tender doesn't come back: by the time anyone spots the gap, the submission deadline has passed and the notice is worth nothing to a bidder.

The Approach: Make Every Fetch Prove What It Saw

Start with what doesn't need scraping. TED's API gives anonymous access to published notices, no key required, so pull the above-threshold side from there. In the UK, Find a Tender has been the central platform where all contracting authorities publish procurement information since 24 February 2025, according to GOV.UK. Check what a central register already covers before you write a parser for a portal it mirrors.

For everything else, three rules carry most of the weight.

1. Define a real answer, including the empty one. A results page carries its column header ("Closing date", in the portal's own words). An honest empty day carries the portal's own empty-state line. Anything else is a gap, not a zero. Put both strings in the request's validate.data.accept list, and an expired-session page fails the request instead of passing as a quiet day. Our engines already refuse to count a verification page they recognise as a success, but no engine knows how every council words its timeout page (that part has to come from you). The match is a plain, case-sensitive substring, so copy the text from the page rather than retyping it. We covered the mechanics when validate rules started deciding success, and the same problem from the time-series side in When a Block Looks Like a Data Point.

2. Check the count the portal prints. Most result lists say how many notices match ("1-20 of 143"). Parse that number and compare it with the rows you stored. If they differ, the run isn't finished, whatever the status codes said.

3. Find the route once, then replay it. Paginated and session-bound portals tend to expect follow-up pages from the same address and the same browser. With a platform like FourA, Auto finds a route that works for page one and returns its session: the proxy ID, the user agent and the cookies from the attempt that worked. Every page after that goes through Single on that exit, which is the production path. Auto is only the path finder.

import math, re, requests

API = "https://api.foura.ai/api"
H = {"X-API-Key": "pk_live_..."}
LIST = "https://tenders.example-council.gov/open?page={}"
PER_PAGE = 20

# A real answer is the results table or the portal's own empty-state line.
# An expired session or a maintenance notice served at 200 is neither.
CHECK = {"data": {"accept": ["Closing date", "No open tenders"]}}

def fetch(endpoint, body):
    r = requests.post(f"{API}/{endpoint}", headers=H, json=body).json()
    if r.get("error"):
        raise RuntimeError(r["error"])  # a gap to refill, not a quiet day
    return r

# Auto finds a route that works and hands back its session...
first = fetch("auto", {"url": LIST.format(1), "validate": CHECK})
proxy, ua = first["session"]["proxy"], first["session"]["userAgent"]
pages = [first["data"]]

# ...and Single replays it on the same exit for every other page.
m = re.search(r"\d+-\d+ of (\d+)", pages[0])
total = int(m.group(1)) if m else 0
for n in range(2, math.ceil(total / PER_PAGE) + 1):
    pages.append(fetch("single", {
        "method": "GET", "url": LIST.format(n), "proxy": proxy,
        "headers": [["User-Agent", ua]], "validate": CHECK,
    })["data"])

If the portal keeps your search state on its server, send its session cookie in the same headers list; Single returns every Set-Cookie header per redirect hop. Tender documents usually sit behind a download link as PDFs, and Single's redirect control and raw buffer mode brings them back as bytes instead of mangled text. One more habit: poll each portal on its own schedule. These are small servers run by small councils, and twenty requests in one burst look a lot worse to them than twenty spread over an afternoon.

Results

  • 25% more tenders captured. GroupBWT, a scraping services firm, reports this for a client monitoring 50+ procurement portals against a manual baseline, "not because more tenders existed, but because they stopped missing them."
  • No scraper for the above-threshold part. TED's notices come from an API that needs no key, so every hour of parser work goes to portals that have no API.
  • Gaps surface while the notice is still open. A page that fails the accept rule raises an error on the day it happens, and the job retries it instead of storing an empty day.
  • One route discovery per portal, not per page. Auto's search happens once; every follow-up page is a Single call on the exit it found.

Only the first bullet is a measured result; the other three describe what the setup changes.

Key Takeaway

In tender monitoring, the most expensive row you can store is "nothing new today", because it's the only one you can't correct later. Prices come back tomorrow. A tender that closed while your scraper was reading an expired-session page doesn't.

So make the job earn its zeros. A list page that can't show you the table, the portal's own empty-state line or a count that matches hasn't told you anything yet, and treating it as an answer is how a bidder ends up learning about a contract from its award notice.