Competitor intelligence — twelve sources, one brief a week
- Problem
- Competitive signal is spread across review sites, app stores, ad libraries, news and video, and reading all of it by hand is a job nobody has time for every week.
- Solution
- Each source scrapes to a dated snapshot, a deterministic diff works out what is new since last time, and only that delta reaches the model.
- Result
- Runs on a schedule once a week and posts its brief into Slack. Built solo at ablefy.
At ablefy, the answer to “what are the others doing” lived in a dozen browser tabs and whoever happened to open them. The useful version of that question is weekly, comparative, and boring to answer by hand — which makes it a pipeline problem rather than a research problem.
This page is about the machine: the lanes it reads, how it decides what changed, and the parts that keep a weekly model job cheap enough to run forever. What the machine finds is internal and stays that way.
The economics are the design. New review counts, score movement and rating mix cost nothing to compute in plain code, so the model is only ever asked why something moved, and every lane runs in brief mode against what is new rather than the whole snapshot.
Two axes, twelve lanes
One command runs the week, and the lanes split along two questions: what customers say, and what the companies do.
On the voice side: reviews from two review platforms, mobile app-store reviews for the product-reliability signal that only shows up on phones, and a community lane for the unfiltered version and for switching talk. On the moves side: website and product-update pages, press mentions, third-party video reviews, the companies' own social posts, their own video channels, live campaigns pulled from the ad library, and the landing-page funnels sitting behind those ads. A twelfth lane runs last and cuts across the others, reading how AI-forward each company is across product, marketing and integrations — last because it folds in the snapshots the earlier lanes just wrote.
Every lane writes into one dated markdown report, and a renderer turns that into the HTML brief someone actually opens.
Deterministic first, model second
Each source scrapes to a timestamped snapshot rather than a live read, and that is what makes the cheap half possible. The diff compares the two newest snapshots and works out what is new in plain Python: group reviews by a stable business identifier that survives a company renaming itself, subtract the previous run's review ids, and report the new count, the trust-score movement, the average rating of the incoming reviews, and how many of them are two stars or below.
Only then is the model called, and only on the rows that are both new and negative — capped for cost, compacted down to rating, title and a truncated body, and asked for three concrete bullets.
Failure handling follows the same split. A scrape that fails skips its own diff, writes the failure into its section of the report, and sets a non-zero exit code so the scheduler alerts. A landing page that comes back thin or blocked is a different thing entirely: the funnel lane records it as blocked and carries on.
The ad is the doorway
The deepest lane goes past the creative. Ad copy says what a company is claiming this week; the landing page says what it is actually selling and how many steps stand between a click and a purchase. The funnel lane reads the fresh ad snapshot, deduplicates the landing URLs behind those ads, drops the ones that point back at the ad platform itself, crawls each real destination to clean markdown at depth zero, and has a model reverse-engineer the hook, the offer, the conversion path in order, and the copy doing the persuading.
Most of that prompt is about what to ignore. Cookie and consent banners, navigation, footers, imprint and privacy pages, bot checks and empty shells are all named as noise — and if the crawl came back as an interstitial, the model is required to label the page as blocked and return no funnel steps rather than inventing a funnel out of a challenge screen. Anything under a few hundred characters is treated as a shell before it ever reaches the model.
The pass this lane was built for: 81 ads across 9 brands, 28 unique landing pages behind them, 28 of 28 reverse-engineered.
Where it lands
The whole thing runs from a scheduled job on a laptop once a week, chosen so the pipeline needs no server access to exist. The runner logs every run, writes the dated report and its rendered brief, then posts the headline summary to a Slack webhook. An optional second path posts the full brief to an automation webhook for routing elsewhere. With neither configured the report simply stays on disk and the run still succeeds.
A separate monthly pass keeps a dated review corpus fresh for a different consumer: my voice-of-customer pipeline, which checks how old its inputs are and flags it when the newest dated review predates the month being reported on. Only the sources that carry a reliable date are worth a monthly pass, so that lane is deliberately narrower than the weekly one.
One lane doubles as a test bench for another project of mine. The review scrape can be routed through Context Wall, my data firewall, behind a flag — which gives the firewall a real recurring workload and gives this pipeline a quality gate on the noisiest source it reads.