Shirley XueAll work
Solo BuildablefyInternal toolLLM pipeline2026

Competitor intelligence — twelve sources, one brief a week

Problem
Competitive signal is spread across review sites, app stores, ad libraries, news and video, and reading all of it by hand is a job nobody has time for every week.
Solution
Each source scrapes to a dated snapshot, a deterministic diff works out what is new since last time, and only that delta reaches the model.
Result
Runs on a schedule once a week and writes one brief, ready to post to Slack. Built solo at ablefy.
Context

At ablefy, the answer to “what are the others doing” lived in a dozen browser tabs and whoever happened to open them. The useful version of that question is weekly, comparative, and boring to answer by hand, which makes it a pipeline problem rather than a research problem.

What the pipeline finds stays internal, so this case covers only how it works.

Sources

What customers say, and what the companies do

One command runs the week, and each source is its own lane.

What customers say: reviews from two review platforms, mobile app-store reviews for the product-reliability signal that only shows up on phones, and a community lane for unfiltered opinion and talk of switching. What the companies do: website and product-update pages, press mentions, third-party video reviews, the companies' own social posts, their own video channels, live campaigns pulled from the ad library, and the landing-page funnels behind those ads. A twelfth lane reads how AI-forward each company is across product, marketing and integrations; it runs last because it folds in the snapshots the earlier lanes just wrote.

Every lane writes into one dated markdown report, and a renderer turns that into the HTML brief someone actually opens.

Diff

Deterministic first, model second

Each source scrapes to a timestamped snapshot rather than a live read, which is what makes the cheap half possible. The diff compares the two newest snapshots in plain Python: group reviews by a stable business identifier that survives a company renaming itself, subtract the previous run's review ids, and report the new count, the trust-score movement, the average rating of the incoming reviews, and how many of them are two stars or below.

Only then is the model asked why something moved, and only on rows that are both new and negative: capped for cost, compacted to rating, title and a truncated body, with a request for three concrete bullets. Every lane runs in brief mode against what is new, not the whole snapshot, which keeps a weekly model job cheap enough to run indefinitely.

Failure handling follows the same split. A scrape that fails skips its own diff, writes the failure into its section of the report, and sets a non-zero exit code so the scheduler alerts. A landing page that comes back thin or blocked is different: the funnel lane records it as blocked and carries on.

Ad funnels

Behind the ads: landing-page funnels

Ad copy says what a company is claiming this week; the landing page says what it is actually selling and how many steps stand between a click and a purchase. The funnel lane reads the fresh ad snapshot, deduplicates the landing URLs behind those ads, drops the ones that point back at the ad platform itself, crawls each real destination to clean markdown at depth zero, and has a model reverse-engineer the hook, the offer, the conversion path in order, and the copy doing the persuading.

Most of that prompt is about what to ignore: cookie and consent banners, navigation, footers, imprint and privacy pages, bot checks and empty shells. If the crawl came back as an interstitial, the model must label the page as blocked and return no funnel steps, rather than invent a funnel from a challenge screen. Anything under a few hundred characters is treated as an empty shell before it ever reaches the model.

The pass this lane was built for: 81 ads across 9 brands, 28 unique landing pages behind them, 28 of 28 reverse-engineered.

Delivery

Where it lands

A scheduled job on a laptop runs it once a week, so the pipeline needs no server access. The runner logs every run, writes the dated report and its rendered brief, then posts the headline summary to a Slack webhook. An optional second path posts the full brief to an automation webhook for routing elsewhere. With neither configured, the report stays on disk and the run still succeeds.

A separate monthly pass keeps a dated review corpus fresh for my voice-of-customer pipeline, which flags when the newest dated review predates the month being reported on. Only sources that carry a reliable date are worth a monthly pass, so that lane is deliberately narrower than the weekly one.

The review scrape can be routed through Context Wall, my data firewall, behind a flag. That gives the firewall a real recurring workload, and gives this pipeline a quality gate on the noisiest source it reads.