Nobody could hear what customers were telling us. So I built the system that could.
Colleagues ask a Slack agent and get answers computed from live data. I designed the taxonomy and the priority formula, fenced where a model may touch the data, and built the checks that run at every step instead of a person reading model output.
One tile is a thousand records. About 237,000 arrive; about 88,000 of them carry an actual customer voice — the filled ones.
The rest is hidden rather than deleted, and every dropped row keeps the reason it was dropped, so the filter itself stays auditable.
One tick is a named problem. All 114 of them, in 30 families and 6 domains.
This is the entire list a model is allowed to choose from. It can pick; it can never invent.
How I built it
Solo, with AI coding agents as the engineering team
- No engineer, no analyst, no tool budget — about four months from first pipeline to production.
- The agents wrote most of the code. Every step of it is checked mechanically before it counts.
- Running it costs about $30 a month, capped in code.
What I built
The taxonomy, the pipeline, the agent, and the harness around them
- A fixed two-level problem list, and the six-label record model underneath it.
- A deterministic priority formula, and a mechanical-first labelling path.
- The agent's tools, and the service that runs it all in production.
What it does now
Used every week — and it says plainly what it does not do
- One typical week: 25 questions from 7 colleagues, every answer graded by an automated judge.
- Asked for a cause the data cannot carry, it says where the evidence stops.
- Synthetic-user personas were tested before being built. On vocabulary the corpus beat them, so I shipped the test instruments instead.
Seven channels, and no answer.
Customer voice reached the company through seven channels, each in its own tool — support tickets in Salesforce, churn and deletion surveys, account-manager calls, CSAT, a semi-annual satisfaction survey, WhatsApp threads, Zoom calls. Each was readable on its own. Nothing joined them.
So the basic product questions had no reliable answer: what are customers struggling with right now, which problems are growing, what is actually driving churn. The answer depended on which channel someone had read last.
It started as research I already owned. In 2025 I had run the same synthesis by hand — fifteen sources, a researcher and me, and a company-wide session on what we found. When the researcher left, the research stayed with me. Nobody commissioned a system: I built one so that a single person could keep answering those questions every month, and it was adopted once it was running.
Ask a question in Slack. The system answers.
A colleague types a question into Slack and gets an answer computed from live data, with the evidence attached — no analyst in the loop, no waiting for a report. The same agent runs in the browser, and a family of reports lives on stable URLs. A Claude Project and a NotebookLM copy used to answer the same questions from a static pack; in October 2026 I retired both, because the agent answers from live data with no manual upload.

No figure is written into the prompt. Every number in an answer comes from one of nine tool calls against live data — including the line about how much data the answer covers, which is derived from MAX(period) at answer time.
The nine tools, named as they are in the code
search_recordsquery_records_by_filteraggregate_metriccompare_periodsfind_seller_cohortget_seller_headcountget_problem_rankingsget_competitor_switchesget_whatsapp_friction
Six decisions shaped the system.
The pipeline, the formula and the harness exist to hold them in production.
- Decision 01
Structure the corpus. Don't summarize it.
Every record is labelled and kept, so the corpus can answer questions nobody has asked yet — and a re-run stays comparable to the run before it.
The path it closed: A model that reads everything and writes up themes produces prose: unqueryable, unrepeatable, unauditable. A summary answers one question, once.
- Decision 02
The model picks from a list. It never invents a category.
Every label is a pick against a fixed vocabulary — 114 problems in 30 families, grouped into 6 domains. Month over month means the same categories, so a trend is a real movement, not a relabelling artifact.
The path it closed: Free clustering is the flexible option — and categories that mutate between runs quietly break every comparison built on them.
- Decision 03
Priority is deterministic code. The model never weighs in.
The ranking has to be explainable to the team that acts on it, and identical when re-run for the same date.
The path it closed: Asking an LLM what matters most returns a different answer every run, and it cannot show its work.
- Decision 04
“Biggest” and “most painful” stay separate.
What is biggest and what hurts most are different questions, so they report side by side rather than blended into one opaque score.
The path it closed: Severity as a score multiplier was proposed, measured, and dropped: volume spans about 30,000× across the ranked set, severity about 2.6× — it barely moved the order while making the score harder to explain.
- Volume~30,000×
- Severity~2.6×
1×10×100×1,000×10,000×100,000×Measured spread across the ranked set, log scale — each tick is ten times the one before it.
- Decision 05
Every number must trace to evidence.
The agent carries no figures in its prompt: if an answer states a number, a tool call against live data produced it. Automated evals — scripted checks that grade the system's real answers — enforce that rule.
- Decision 06
The system surfaces evidence. It does not decide.
Asked why customers churn, it answers with what the data supports and says plainly where the link stops — the strongest churn reason in the corpus is business closure and life change, which no product decision reaches. Ranking bands are claimed (the top five beat eleven to fifteen); finer resolution inside a band is refused. The judgement about what to build stays with the team that owns the roadmap.
The path it closed: An agent that answers every question confidently is the easier product and the one nobody can act on: the first time it asserts a cause the data cannot carry, every other answer becomes suspect.
I almost built synthetic users. Then I tested them.
The most tempting idea in the project was synthetic users: personas grounded in each customer's real voice history, something a team could question like a panel. Instead of building it, I designed four experiments to test whether the personas could be trusted first.
The experiments came back split, and the split was the finding. Personas carry register and tone convincingly — but on vocabulary, what customers actually call things, direct retrieval from the corpus beat them decisively.
So the verdict says don't build — recorded right next to the go decision, where the next person tempted by the same idea will find it. The experiments still shipped, as two command-line instruments other projects now use: one runs a persona panel for reactions and misreadings, treated as hypotheses; one answers vocabulary questions straight from the corpus.
A mechanical check at every step, reported weekly.
I build with AI in domains I cannot fully audit by eye. So wherever a person would otherwise have to trust model output, a mechanical check runs instead — ten of them, listed in the appendix.
Every production question becomes a trace. Every Monday a judge scores the week's traces for faithfulness, over-claiming, scope honesty and language match — whether the person acting on an answer can trust it, not whether it reads well — and posts its findings in Slack with its reasoning. One recent week: 25 questions from 7 askers, 6 of 24 substantive answers flagged. A flagged answer becomes the next fix; a user's thumbs-up or thumbs-down lands on the exact trace; prompt or tool changes batch into one full eval before deploy.
One rule came out of building it: a fabrication judge that cannot see the source evidence flags grounded specifics just for being specific. Any grounding check has to show the judge the evidence, not only the answer.
Evidence that changed a decision.
The clearest example sits in the pricing case. When a regulation retired one plan type and the requirements document proposed three better names on the same picker, the 205 seller call transcripts in this corpus showed that the word “installment” never occurs — sellers and the staff explaining it say “payment plan”. That finding backed replacing the plan-type picker with two plain questions instead of relabelling it.
Appendix · how the system worksArchitecture, data model, priority engine, LLM discipline, and the full harness — five parts.How the system holds the six decisions in production.
Seven channels in, one catalog out.
Seven channels carry customer voice into the pipeline, most of them monthly. Commercial exports — the customer census, the ICP roster (the business's own list of ideal-profile customers), the revenue-churn list — are consumed and never recomputed; those numbers stay business-owned.
The pipeline is six stages, and the order is the point: the cheap deterministic work happens first, so the expensive work only ever runs on rows that survived it.
Storage is a few plain files, not a warehouse: the voice lake lives as Parquet files on S3, with the aggregate tables beside it, and every table is exposed as a DuckDB view reading those files directly.
- Support ticketsSalesforce · monthly
- Churn & deletion surveysUserflow + CRM · monthly
- Account-manager calls
- CSATmonthly
- Platform satisfaction surveysemi-annual
- WhatsApp threadsfixed historical corpus
- Zoom callsnewest · monthly
- 01Ingest
- 02Clean · filter · redact
- 03Label
- 04Join + churn state
- 05Rank
- 06Publish
- One voice lake — every record row-level, aggregates beside it.
- Queried as views; the view name is the contract.
- A monthly sync is what makes new data live.
- Ask agentin Slack and in the browser
- Report familyDashboard, Brief, Reference — on stable URLs
- Knowledge packsbuilt monthly; the Claude Project and NotebookLM copies retired Oct 2026
- 01IngestPer-channel specs. Normalises customer IDs and German number and date formats, so everything downstream shares one basis.
- 02Clean · filter · redactThree steps: structural tidy and dedupe; a noise filter where every dropped row keeps a reason flag; personal-data (PII) redaction on the survivors only, channel-aware so call transcripts keep the customer's own words.
- 03LabelSix independent labels per record, each a pick from a closed list.
- 04Join + churn stateJoined by customer ID: ICP membership, churn reason and destination, account state, revenue, tenure.
- 05RankThe priority formula, run as of a date. Same date in, same ranking out.
- 06PublishHard gate: the month must be complete and every derived table fresh, or nothing publishes. Enforced in code, not on a checklist.
One record, six independent labels.
Real feedback carries several signals at once, so a record is tagged along six independent dimensions rather than squeezed into one label — each of them a pick from its own closed list.
Domain-to-team is derived, not labelled — a strict function of the problem through one editable map, so a re-org edits the map instead of re-tagging the corpus.
I love X but I'm leaving for Y because you lack Z.
| Label | Closed vocabulary | Written onto this record |
|---|---|---|
| problem family → problem | a fixed two-level list: 114 problems in 30 families, grouped into 6 domains | the capability named in “you lack Z”, classified into the list — never free-clustered |
| need / impact | 8 values, including abstain | what was needed and how blocked they were, read from “you lack Z” |
| type | noise · product · operational | product |
| journey stage | 10 stages, awareness → churn, plus migration-in · multi-label | churn, from “I'm leaving” |
| flags | ux friction · plan-gated | set from the text |
| voice source | customer · end buyer · provider or outbound · system | customer — this is the noise guard |
- Labelproblem family → problemClosed vocabularya fixed two-level list: 114 problems in 30 families, grouped into 6 domainsWritten onto this recordthe capability named in “you lack Z”, classified into the list — never free-clustered
- Labelneed / impactClosed vocabulary8 values, including abstainWritten onto this recordwhat was needed and how blocked they were, read from “you lack Z”
- LabeltypeClosed vocabularynoise · product · operationalWritten onto this recordproduct
- Labeljourney stageClosed vocabulary10 stages, awareness → churn, plus migration-in · multi-labelWritten onto this recordchurn, from “I'm leaving”
- LabelflagsClosed vocabularyux friction · plan-gatedWritten onto this recordset from the text
- Labelvoice sourceClosed vocabularycustomer · end buyer · provider or outbound · systemWritten onto this recordcustomer — this is the noise guard
The formula, term by term.
A mention halves every 45 days, so the order follows what customers feel now rather than what they felt last year. Revenue is damped twice — log, then square root — so 100× the revenue is about 1.3× the weight: reach drives the order and revenue only tilts it.
Reach is distinct customers, each counted by their most recent mention, so one power user filing fifty tickets still counts as one. High-value accounts and everyone else are ranked apart, so ICP customers cannot bury the rest. Channels combine only when they share a customer-ID basis and an overlapping window; calls without an ID stay a separate descriptive view. A dormancy check and a pipeline-break guard keep a data gap from masquerading as a trend.
score = reach (recency-weighted) × √(log revenue)
- reach
- Distinct customers, each counted by their most recent mention. One power user filing fifty tickets still counts as one.
- recency-weighted
- A mention halves every 45 days.
- revenue, damped
- Revenue damped twice, log then square root: 100× the revenue is about 1.3× the weight.
Mechanical first. One paid call per row.
There are exactly three fenced uses for a model here: categorisation, extraction, and the Ask agent; the weekly judge only grades answers. Everywhere else, deterministic code does the work. Labelling one record is free mechanical work, then one structured call against the fixed canonical list, then free validation — the expensive step can only choose from a list it was handed.
Two models, chosen by bake-off, both behind one proxied endpoint — spend capped in code, no keys in the build. Every call goes through one shared wrapper that caches prompts and routes every trace into a single Langfuse project — which is what makes the harness possible at all.
| Step | What runs | Cost |
|---|---|---|
| Mechanical prefilters and exact maps | Deterministic code does everything it can before a model is asked anything. | free |
| One structured call per record | Against the fixed canonical list — a closed vocabulary. The model picks from the list; it cannot invent a category. One corrective retry, then a defined fallback. | $0.0027 |
| Validation and derivation | Deterministic again: the derived fields are computed, not asked for. | free |
- StepMechanical prefilters and exact mapsWhat runsDeterministic code does everything it can before a model is asked anything.Costfree
- StepOne structured call per recordWhat runsAgainst the fixed canonical list — a closed vocabulary. The model picks from the list; it cannot invent a category. One corrective retry, then a defined fallback.Cost$0.0027
- StepValidation and derivationWhat runsDeterministic again: the derived fields are computed, not asked for.Costfree
| Model | Where it runs | Why this one |
|---|---|---|
| Gemini 2.5 Flash | bulk categorisation | won the bake-off for this job |
| Claude Sonnet 4.5 (Bedrock) | the agent, churn-reason extraction, signal-kind labelling | cheaper models fell below the accuracy bar |
- ModelGemini 2.5 FlashWhere it runsbulk categorisationWhy this onewon the bake-off for this job
- ModelClaude Sonnet 4.5 (Bedrock)Where it runsthe agent, churn-reason extraction, signal-kind labellingWhy this onecheaper models fell below the accuracy bar
Every check, what it holds, and when it runs.
The ten mechanical checks behind section 05, as a table.
| Check | What it holds | When |
|---|---|---|
| Calibration gate | A classifier may not run at full scale until it agrees with a human-labelled calibration set on at least 90% of rows. Results are cached, so a re-run never silently relabels. The gate is the rule, not a score — every gold row carries who reviewed it, edge cases are excluded from the primary metric, and the accuracy that matters is measured on the production run rather than on the set the prompt was written against. | before any full-scale run |
| Closed vocabularies | Every label is a pick from a fixed list. The model cannot invent a category. | every row |
| Grounded answers | The agent never recites a figure written into its instructions. Every number comes from a tool call against live data — including the coverage line, derived from MAX(period). | eval-enforced |
| Gold sets, never edited to fit | The eval set is human-verified and fixed. When an answer disagrees with it, the answer is what changes. The labelling gold set is 40 hand-labelled rows, and the prompt was tuned on them — so it grades the model on cases it was built for, and real out-of-distribution accuracy runs a few points lower. Production accuracy is measured separately, by stratified spot-checks of 15 to 25 rows against the actual run. | every eval run |
| Live reference anchors | Anchors recompute each run, so normal monthly data movement does not break the test set. | every eval run |
| Taxonomy drift monitor | Watches the taxonomy for drift. | ongoing |
| Noise hidden, not deleted | Filtered rows keep a reason flag and stay in the lake; noise is hidden at query time, so filter quality stays auditable. | at query time |
| Weekly judge | Scores live answers for faithfulness, over-claiming, scope honesty and language match. A thumbs-up or thumbs-down from a user lands as a score on the exact trace it came from. | Mondays, 07:00 UTC |
| Deploy-gate eval | Prompt and tool changes batch into one full run against the fixed gold set. A change counts as done only when the deployed system is confirmed to serve the new answer. | per deploy |
| Publish gate | The month must be complete and every derived table fresh, or nothing publishes. | every monthly publish |
- CheckCalibration gateWhat it holdsA classifier may not run at full scale until it agrees with a human-labelled calibration set on at least 90% of rows. Results are cached, so a re-run never silently relabels. The gate is the rule, not a score — every gold row carries who reviewed it, edge cases are excluded from the primary metric, and the accuracy that matters is measured on the production run rather than on the set the prompt was written against.Whenbefore any full-scale run
- CheckClosed vocabulariesWhat it holdsEvery label is a pick from a fixed list. The model cannot invent a category.Whenevery row
- CheckGrounded answersWhat it holdsThe agent never recites a figure written into its instructions. Every number comes from a tool call against live data — including the coverage line, derived from MAX(period).Wheneval-enforced
- CheckGold sets, never edited to fitWhat it holdsThe eval set is human-verified and fixed. When an answer disagrees with it, the answer is what changes. The labelling gold set is 40 hand-labelled rows, and the prompt was tuned on them — so it grades the model on cases it was built for, and real out-of-distribution accuracy runs a few points lower. Production accuracy is measured separately, by stratified spot-checks of 15 to 25 rows against the actual run.Whenevery eval run
- CheckLive reference anchorsWhat it holdsAnchors recompute each run, so normal monthly data movement does not break the test set.Whenevery eval run
- CheckTaxonomy drift monitorWhat it holdsWatches the taxonomy for drift.Whenongoing
- CheckNoise hidden, not deletedWhat it holdsFiltered rows keep a reason flag and stay in the lake; noise is hidden at query time, so filter quality stays auditable.Whenat query time
- CheckWeekly judgeWhat it holdsScores live answers for faithfulness, over-claiming, scope honesty and language match. A thumbs-up or thumbs-down from a user lands as a score on the exact trace it came from.WhenMondays, 07:00 UTC
- CheckDeploy-gate evalWhat it holdsPrompt and tool changes batch into one full run against the fixed gold set. A change counts as done only when the deployed system is confirmed to serve the new answer.Whenper deploy
- CheckPublish gateWhat it holdsThe month must be complete and every derived table fresh, or nothing publishes.Whenevery monthly publish
What this system does not do.
Each one is a trade-off I chose, with its reason.
- Labelling covers ICP sellers and roughly the last twelve months — a cut chosen on the measurement: with a 45-day half-life, that scope carries about 99.3% of the recency signal at roughly a quarter of the cost. Older and non-ICP voice is answered from a thinner pool, and the coverage fills in over time.
- The no-login churn rule is switched off. The login feed lags real activity, so it is better to under-count churn than to mis-flag active customers. It comes back when a reliable activity feed exists.
- need / impact is the softest of the six labels: it often needs product state the model cannot see, so it is used for coarse routing, never as a fine score.
- The WhatsApp corpus will not grow. Live capture is off, and the corpus stays a fixed historical set.
- Churn rate, ICP membership and the customer census are business-owned. The system uses them and never recomputes them.
The taxonomy and the domain-to-team map are employer-specific by design. The patterns are portable.
