Nobody could hear what customers were telling us. So I built the system that could.

Customer voice reached the company through seven channels, each in its own tool — support tickets in Salesforce, churn and deletion surveys, account-manager calls, CSAT, a semi-annual satisfaction survey, WhatsApp threads, Zoom calls. Each was readable on its own. Nothing joined them.

So the basic product questions had no reliable answer: what are customers struggling with right now, which problems are growing, what is actually driving churn. The answer depended on which channel someone had read last. Fixing that meant building infrastructure — with no engineer, no analyst and no tool budget.

A stakeholder types a question into Slack and gets an answer computed from live data, evidence attached — no analyst in the loop, no waiting for a report. The same agent runs in the browser over one backend; a report family lives on stable URLs; knowledge packs mirror into a Claude Project and NotebookLM. Three front doors, one catalog behind them.

The catalog is the contract: every record row-level, aggregates beside it, queried as views — the view name is the only thing a front door needs to know. A monthly sync is what makes new data live, and the agent works it with nine tools.

Six decisions shaped the system. The pipeline, the formula and the harness are what it takes to hold them in production.

The most tempting idea in the project was synthetic users: personas grounded in each customer's real voice history, something a team could question like a panel. Instead of building it, I designed four experiments to test whether the personas could be trusted first.

The experiments came back split, and the split was the finding. Personas carry register and tone convincingly — but on vocabulary, what customers actually call things, direct retrieval from the corpus beat them decisively.

So the verdict says don't build — recorded right next to the go decision, where the next person tempted by the same idea will find it. The experiments still shipped, as two command-line instruments other projects now use: one runs a persona panel where tone is what matters, one answers vocabulary questions straight from the corpus.

Seven channels carry customer voice into the pipeline, most of them monthly. Commercial exports — the customer census, the ICP roster (the business's own list of ideal-profile customers), the revenue-churn list — are consumed and never recomputed; those numbers stay business-owned.

The pipeline is six stages, and the order is the point: the cheap deterministic work happens first, so the expensive work only ever runs on rows that survived it.

Storage is a few plain files, not a warehouse: the voice lake lives as Parquet files on S3, with the aggregate tables beside it, and every table is exposed as a DuckDB view reading those files directly.

Real feedback carries several signals at once, so a record is tagged along six independent dimensions rather than squeezed into one label — each of them a pick from its own closed list.

Domain-to-team is derived, not labelled — a strict function of the problem through one editable map, so a re-org edits the map instead of re-tagging the corpus.

A mention halves every 45 days, so the order follows what customers feel now rather than what they felt last year. Revenue is damped twice — log, then square root — so 100× the revenue is about 1.3× the weight: reach drives the order and revenue only tilts it.

Reach is distinct customers, each counted by their most recent mention, so one power user filing fifty tickets still counts as one. High-value accounts and everyone else are ranked apart, so ICP customers cannot bury the rest. Channels combine only when they share a customer-ID basis and an overlapping window; calls without an ID stay a separate descriptive view. A dormancy check and a pipeline-break guard keep a data gap from masquerading as a trend.

There are exactly three fenced uses for a model here: categorisation, extraction, and the Ask agent. Everywhere else, deterministic code does the work. Labelling one record is free mechanical work, then one structured call against the fixed canonical list, then free validation — the expensive step can only choose from a list it was handed.

Two models, chosen by bake-off, both behind one proxied endpoint — spend capped in code, no keys in the build. Every call goes through one shared wrapper that caches prompts and routes every trace into a single Langfuse project — which is what makes the harness possible at all.

I am not an engineer by background, and I build with AI in domains I cannot fully audit by eye. So the system checks itself at every step instead of me reading model output and hoping. The calibration gate, the closed vocabularies, the grounded-answer rule, the gold sets, the live reference anchors, the drift monitor and the deploy-gate eval are one strategy, not seven features: put a mechanical check wherever a human would otherwise have to trust something.

The loop is closed at both ends. Every production question becomes a trace; a judge scores those traces weekly for faithfulness, over-claiming, scope honesty and language match — not whether an answer reads well, but whether the person acting on it can trust it. A thumbs-up or thumbs-down from a user lands on the exact trace it came from, and prompt or tool changes batch into one full eval at the deploy gate. A change counts as done only when the deployed system is confirmed to serve the new answer.

One rule came out of building it, found mid-run: a fabrication judge with no access to the source evidence is unreliable. It over-flags legitimately grounded specifics purely for being specific. Any grounding check has to show the judge the evidence, not just the answer.

Every Monday the judge posts what it found: how many questions were asked, by how many people, and which answers failed on faithfulness, over-claiming, scope honesty or language match — with its reasoning attached. One recent week: 25 questions from 7 distinct askers, 6 of 24 substantive answers flagged. A flagged answer becomes the next fix.

Hallucination and citation fixes ship as separate changes, each named for the bug it closed; the deploy path itself is code, so applying it cannot drift from what is written. The work around the fixes is where the judgment shows: