Context Wall — read the content, not the status code
- Problem
- Web scraping fails about 5% of the time, and the failure often arrives as HTTP 200 with a bot wall in the body. The shape check passes, the agent reasons over it, and the tokens are already spent.
- Solution
- One MCP tool the agent calls instead of the scraper. Tier 1 reads every row as it arrives; Tier 2 judges intent match on a sample. Whichever blocks first aborts the run.
- Result
- 3rd place at “Agents that pay”, Berlin. Runs offline on fixtures or live against a deployed Apify actor that fetches a real URL.
An agent buying web data has no way to tell a good row from a poisoned one. It gets JSON, the JSON has the right keys, and it starts reasoning. The failure mode is quiet: the agent answers confidently from a Cloudflare page, and the buyer pays twice — once for the scrape, once for the tokens spent reading it.
Context Wall is a standalone MCP server — the plug-in protocol AI assistants use to call tools — exposing a single tool, scrape_validated. The agent hands over its intent in plain language and the fields it needs; the firewall drives the upstream scraper, checks the stream, and returns either clean data or a structured rejection saying why not. Built for the “Agents that pay” hackathon in Berlin, where it took 3rd place, and kept as an open repo afterwards.
- Read the content. The status code is the one signal a block page can forge, and it does.
- Fail fast and stop the meter: the first blocking verdict aborts the upstream cloud run, so the job stops before it finishes billing.
- A missing judge is never a pass. With no LLM key, Tier 2 falls back to a heuristic instead of waving the data through.
A success that is really a wall
This is a real row a scrape brought back from homedepot.com: status code 200, an empty title, and a body reading “Powered and protected by Privacy”. Valid JSON, valid shape, correct keys, zero data. A naive check passes it because there is no error to notice — the anti-bot vendor answers with success and puts the wall in the body.
The louder version exists too: yellowpages.com returns a 403 with the classic Cloudflare page. Both are caught the same way, by reading the content.
Two things go wrong when a row like that lands. The agent reasons over block-page text and answers from it, and the token budget is spent on junk it also paid a scraper to fetch. There is a quieter third case with the same cost: data that is real but wrong — you asked for restaurants with delivery and got dine-in only.
Judge the stream, not the buffer
Buffering the whole result and then checking it means paying for the whole result. The firewall checks rows as they arrive, with one abort controller governing the run.
- 01
The agent calls one tool
scrape_validated takes the agent's intent in natural language plus the fields the result must carry, and stands in for the raw scraper call.
- 02
Tier 1 runs on every row
Pure TypeScript, no network, targeting under two milliseconds: a regex blocklist over the stringified item, an emptiness check, and a required-field check. It runs the instant a row arrives, so the first poisoned row is the last one processed.
- 03
Tier 2 runs concurrently
One LLM call on the first few buffered rows, in flight while the rest of the stream keeps arriving, asking whether the data matches what the agent actually wanted.
- 04
Either tier trips the breaker
The first blocking verdict aborts the shared controller: the local reader stops and the cloud container is killed. In the hard-fail demo, zero rows are delivered and the job stops after roughly one of twelve.
The blocklist is a field record
Tier 1 started with the classics — Cloudflare, CAPTCHA, Access Denied, “Attention Required” — and grew from pages that real scrapes came back with. It now carries “Ray ID”, which every Cloudflare error page prints; “Access to this page has been denied” and “Pardon our interruption” from two different bot-management vendors; “Checking your browser”; vendor names that leak into their own interstitials; and “Powered and protected by Privacy”, the wall served at 200 where the body is the only tell.
Because the match runs over the item's content rather than its metadata, one rule covers a 403 with a Cloudflare page and a 200 with an Akamai one. A row that stringifies to nothing is blocked as well, and a caller that names required fields turns a missing value into a block rather than a silent gap downstream.
The verdict that comes back is specific enough to act on: the tier that fired, the reason, the phrase that matched, and the row index it matched at. The terminal dashboard pairs that with rows produced against rows total — the proof the breaker fired early — and an estimate of the tokens kept out of the context window.
A standalone gateway
Context Wall is its own MCP server rather than a patched copy of a vendor's. It drives the upstream scraper behind a single Scraper interface — one provider today, another tomorrow — and the agent's call never changes.
Three ways to watch it run. Offline fixtures stream three fixed scenarios row by row and honour the same abort signal as the cloud client, which makes a stage demo deterministic and free; the hard fixture is perfectly shaped JSON whose values are block-page text, so it passes a schema check and fails the firewall. A live mode drives a deployed actor that actually fetches a URL, where an anti-bot site hands back a genuine block page. And a simulated buyer agent talks to the server over real MCP and prints its own decision: circuit broken, reason given, do not ingest, do not pay.