Case study · Voice of Customer · ablefy · internal business figures redacted

Nobody could hear what customers were telling us. So I built the system that could.

Colleagues ask a Slack agent and get answers computed from live data. I designed the taxonomy and the priority formula, fenced where a model may touch the data, and built the checks that run at every step instead of a person reading model output.

What arrives · what it has to become

One tile is a thousand records. About 237,000 arrive; about 88,000 of them carry an actual customer voice — the filled ones.

The rest is hidden rather than deleted, and every dropped row keeps the reason it was dropped, so the filter itself stays auditable.

One tick is a named problem. All 114 of them, in 30 families and 6 domains.

This is the entire list a model is allowed to choose from. It can pick; it can never invent.

How I built it

Solo, with AI coding agents as the engineering team

  • No engineer, no analyst, no tool budget — about four months from first pipeline to production.
  • The agents wrote most of the code. Every step of it is checked mechanically before it counts.
  • Running it costs about $30 a month, capped in code.

What I built

The taxonomy, the pipeline, the agent, and the harness around them

  • A fixed two-level problem list, and the six-label record model underneath it.
  • A deterministic priority formula, and a mechanical-first labelling path.
  • The agent's tools, and the service that runs it all in production.

What it does now

Used every week — and it says plainly what it does not do

  • One typical week: 25 questions from 7 colleagues, every answer graded by an automated judge.
  • Asked for a cause the data cannot carry, it says where the evidence stops.
  • Synthetic-user personas were tested before being built. On vocabulary the corpus beat them, so I shipped the test instruments instead.
The problem

Seven channels, and no answer.

Customer voice reached the company through seven channels, each in its own tool — support tickets in Salesforce, churn and deletion surveys, account-manager calls, CSAT, a semi-annual satisfaction survey, WhatsApp threads, Zoom calls. Each was readable on its own. Nothing joined them.

So the basic product questions had no reliable answer: what are customers struggling with right now, which problems are growing, what is actually driving churn. The answer depended on which channel someone had read last.

It started as research I already owned. In 2025 I had run the same synthesis by hand — fifteen sources, a researcher and me, and a company-wide session on what we found. When the researcher left, the research stayed with me. Nobody commissioned a system: I built one so that a single person could keep answering those questions every month, and it was adopted once it was running.

What shipped

Ask a question in Slack. The system answers.

A colleague types a question into Slack and gets an answer computed from live data, with the evidence attached — no analyst in the loop, no waiting for a report. The same agent runs in the browser, and a family of reports lives on stable URLs. A Claude Project and a NotebookLM copy used to answer the same questions from a static pack; in October 2026 I retired both, because the agent answers from live data with no manual upload.

A real exchange with the Ask agent in Slack: asked how ablefy compares to competitors on payouts, it answers with public competitor review quotes, internal question themes, an explicit warning that the two sides' numbers are not comparable, and a thumbs-up-or-down feedback row
A production question and its answer, reconstructed — trimmed at the marked elision, internal counts removed. The warning about non-comparable numbers is the agent's own. Click for full size.
The rule the agent runs under

No figure is written into the prompt. Every number in an answer comes from one of nine tool calls against live data — including the line about how much data the answer covers, which is derived from MAX(period) at answer time.

The nine tools, named as they are in the code
  • search_records
  • query_records_by_filter
  • aggregate_metric
  • compare_periods
  • find_seller_cohort
  • get_seller_headcount
  • get_problem_rankings
  • get_competitor_switches
  • get_whatsapp_friction
Design decisions

Six decisions shaped the system.

The pipeline, the formula and the harness exist to hold them in production.

  1. Decision 01

    Structure the corpus. Don't summarize it.

    Every record is labelled and kept, so the corpus can answer questions nobody has asked yet — and a re-run stays comparable to the run before it.

    The path it closed: A model that reads everything and writes up themes produces prose: unqueryable, unrepeatable, unauditable. A summary answers one question, once.

  2. Decision 02

    The model picks from a list. It never invents a category.

    Every label is a pick against a fixed vocabulary — 114 problems in 30 families, grouped into 6 domains. Month over month means the same categories, so a trend is a real movement, not a relabelling artifact.

    The path it closed: Free clustering is the flexible option — and categories that mutate between runs quietly break every comparison built on them.

  3. Decision 03

    Priority is deterministic code. The model never weighs in.

    The ranking has to be explainable to the team that acts on it, and identical when re-run for the same date.

    The path it closed: Asking an LLM what matters most returns a different answer every run, and it cannot show its work.

  4. Decision 04

    “Biggest” and “most painful” stay separate.

    What is biggest and what hurts most are different questions, so they report side by side rather than blended into one opaque score.

    The path it closed: Severity as a score multiplier was proposed, measured, and dropped: volume spans about 30,000× across the ranked set, severity about 2.6× — it barely moved the order while making the score harder to explain.

    • Volume~30,000×
    • Severity~2.6×
    1×10×100×1,000×10,000×100,000×

    Measured spread across the ranked set, log scale — each tick is ten times the one before it.

  5. Decision 05

    Every number must trace to evidence.

    The agent carries no figures in its prompt: if an answer states a number, a tool call against live data produced it. Automated evals — scripted checks that grade the system's real answers — enforce that rule.

  6. Decision 06

    The system surfaces evidence. It does not decide.

    Asked why customers churn, it answers with what the data supports and says plainly where the link stops — the strongest churn reason in the corpus is business closure and life change, which no product decision reaches. Ranking bands are claimed (the top five beat eleven to fifteen); finer resolution inside a band is refused. The judgement about what to build stays with the team that owns the roadmap.

    The path it closed: An agent that answers every question confidently is the easier product and the one nobody can act on: the first time it asserts a cause the data cannot carry, every other answer becomes suspect.

A reversal

I almost built synthetic users. Then I tested them.

The most tempting idea in the project was synthetic users: personas grounded in each customer's real voice history, something a team could question like a panel. Instead of building it, I designed four experiments to test whether the personas could be trusted first.

The experiments came back split, and the split was the finding. Personas carry register and tone convincingly — but on vocabulary, what customers actually call things, direct retrieval from the corpus beat them decisively.

So the verdict says don't build — recorded right next to the go decision, where the next person tempted by the same idea will find it. The experiments still shipped, as two command-line instruments other projects now use: one runs a persona panel for reactions and misreadings, treated as hypotheses; one answers vocabulary questions straight from the corpus.

The harness

A mechanical check at every step, reported weekly.

I build with AI in domains I cannot fully audit by eye. So wherever a person would otherwise have to trust model output, a mechanical check runs instead — ten of them, listed in the appendix.

Every production question becomes a trace. Every Monday a judge scores the week's traces for faithfulness, over-claiming, scope honesty and language match — whether the person acting on an answer can trust it, not whether it reads well — and posts its findings in Slack with its reasoning. One recent week: 25 questions from 7 askers, 6 of 24 substantive answers flagged. A flagged answer becomes the next fix; a user's thumbs-up or thumbs-down lands on the exact trace; prompt or tool changes batch into one full eval before deploy.

One rule came out of building it: a fabrication judge that cannot see the source evidence flags grounded specifics just for being specific. Any grounding check has to show the judge the evidence, not only the answer.

The weekly Ask-agent quality digest in Slack: 25 questions from 7 distinct askers, 6 of 24 substantive answers flagged, and two judge verdicts with their reasoning — internal names redacted
A real weekly digest, reconstructed for this page — trimmed, internal names and links redacted (the grey bars). Click for full size.
In use

Evidence that changed a decision.

The clearest example sits in the pricing case. When a regulation retired one plan type and the requirements document proposed three better names on the same picker, the 205 seller call transcripts in this corpus showed that the word “installment” never occurs — sellers and the staff explaining it say “payment plan”. That finding backed replacing the plan-type picker with two plain questions instead of relabelling it.

The pricing case, where that finding landed →

Appendix · how the system worksArchitecture, data model, priority engine, LLM discipline, and the full harness — five parts.How the system holds the six decisions in production.
Architecture

Seven channels in, one catalog out.

Seven channels carry customer voice into the pipeline, most of them monthly. Commercial exports — the customer census, the ICP roster (the business's own list of ideal-profile customers), the revenue-churn list — are consumed and never recomputed; those numbers stay business-owned.

The pipeline is six stages, and the order is the point: the cheap deterministic work happens first, so the expensive work only ever runs on rows that survived it.

Storage is a few plain files, not a warehouse: the voice lake lives as Parquet files on S3, with the aggregate tables beside it, and every table is exposed as a DuckDB view reading those files directly.

Seven channels · six stages · one catalog · three front doors
Channels in
  • Support ticketsSalesforce · monthly
  • Churn & deletion surveysUserflow + CRM · monthly
  • Account-manager calls
  • CSATmonthly
  • Platform satisfaction surveysemi-annual
  • WhatsApp threadsfixed historical corpus
  • Zoom callsnewest · monthly
Pipeline
  1. 01Ingest
  2. 02Clean · filter · redact
  3. 03Label
  4. 04Join + churn state
  5. 05Rank
  6. 06Publish
One catalog
  • One voice lake — every record row-level, aggregates beside it.
  • Queried as views; the view name is the contract.
  • A monthly sync is what makes new data live.
Front doors
  • Ask agentin Slack and in the browser
  • Report familyDashboard, Brief, Reference — on stable URLs
  • Knowledge packsbuilt monthly; the Claude Project and NotebookLM copies retired Oct 2026
The six stages
  1. 01
    IngestPer-channel specs. Normalises customer IDs and German number and date formats, so everything downstream shares one basis.
  2. 02
    Clean · filter · redactThree steps: structural tidy and dedupe; a noise filter where every dropped row keeps a reason flag; personal-data (PII) redaction on the survivors only, channel-aware so call transcripts keep the customer's own words.
  3. 03
    LabelSix independent labels per record, each a pick from a closed list.
  4. 04
    Join + churn stateJoined by customer ID: ICP membership, churn reason and destination, account state, revenue, tenure.
  5. 05
    RankThe priority formula, run as of a date. Same date in, same ranking out.
  6. 06
    PublishHard gate: the month must be complete and every derived table fresh, or nothing publishes. Enforced in code, not on a checklist.
Data model

One record, six independent labels.

Real feedback carries several signals at once, so a record is tagged along six independent dimensions rather than squeezed into one label — each of them a pick from its own closed list.

Domain-to-team is derived, not labelled — a strict function of the problem through one editable map, so a re-org edits the map instead of re-tagging the corpus.

One record · six labels

I love X but I'm leaving for Y because you lack Z.

  • Label
    problem family → problem
    Closed vocabulary
    a fixed two-level list: 114 problems in 30 families, grouped into 6 domains
    Written onto this record
    the capability named in “you lack Z”, classified into the list — never free-clustered
  • Label
    need / impact
    Closed vocabulary
    8 values, including abstain
    Written onto this record
    what was needed and how blocked they were, read from “you lack Z”
  • Label
    type
    Closed vocabulary
    noise · product · operational
    Written onto this record
    product
  • Label
    journey stage
    Closed vocabulary
    10 stages, awareness → churn, plus migration-in · multi-label
    Written onto this record
    churn, from “I'm leaving”
  • Label
    flags
    Closed vocabulary
    ux friction · plan-gated
    Written onto this record
    set from the text
  • Label
    voice source
    Closed vocabulary
    customer · end buyer · provider or outbound · system
    Written onto this record
    customer — this is the noise guard
“Y” is not a label. A leaver who names a competitor gets that platform recorded as a destination, and a reason only if they actually stated a driver — switching is a result, not a reason. Churn reasons are their own axis: a fixed nine-value vocabulary, kept apart from product problems.Derived, not labelled: domain → team is a strict function of the problem through one editable map, so a re-org edits the map instead of the labels. Severity is reported as a separate facet. Joined by customer ID: ICP membership, churn reason and destination, account state, revenue, tenure.
Priority engine

The formula, term by term.

A mention halves every 45 days, so the order follows what customers feel now rather than what they felt last year. Revenue is damped twice — log, then square root — so 100× the revenue is about 1.3× the weight: reach drives the order and revenue only tilts it.

Reach is distinct customers, each counted by their most recent mention, so one power user filing fifty tickets still counts as one. High-value accounts and everyone else are ranked apart, so ICP customers cannot bury the rest. Channels combine only when they share a customer-ID basis and an overlapping window; calls without an ID stay a separate descriptive view. A dormancy check and a pipeline-break guard keep a data gap from masquerading as a trend.

The formula

score = reach (recency-weighted) × √(log revenue)

reach
Distinct customers, each counted by their most recent mention. One power user filing fifty tickets still counts as one.
recency-weighted
A mention halves every 45 days.
revenue, damped
Revenue damped twice, log then square root: 100× the revenue is about 1.3× the weight.
Pure and deterministic: the same as-of date in, the same ranking out.
LLM discipline

Mechanical first. One paid call per row.

There are exactly three fenced uses for a model here: categorisation, extraction, and the Ask agent; the weekly judge only grades answers. Everywhere else, deterministic code does the work. Labelling one record is free mechanical work, then one structured call against the fixed canonical list, then free validation — the expensive step can only choose from a list it was handed.

Two models, chosen by bake-off, both behind one proxied endpoint — spend capped in code, no keys in the build. Every call goes through one shared wrapper that caches prompts and routes every trace into a single Langfuse project — which is what makes the harness possible at all.

Labelling one record
  • Step
    Mechanical prefilters and exact maps
    What runs
    Deterministic code does everything it can before a model is asked anything.
    Cost
    free
  • Step
    One structured call per record
    What runs
    Against the fixed canonical list — a closed vocabulary. The model picks from the list; it cannot invent a category. One corrective retry, then a defined fallback.
    Cost
    $0.0027
  • Step
    Validation and derivation
    What runs
    Deterministic again: the derived fields are computed, not asked for.
    Cost
    free
One paid call per row, and it can only choose from a list it was handed. One corrective retry, then a defined fallback. Results are cached, so a re-run never silently relabels.
Two models, chosen by bake-off
  • Model
    Gemini 2.5 Flash
    Where it runs
    bulk categorisation
    Why this one
    won the bake-off for this job
  • Model
    Claude Sonnet 4.5 (Bedrock)
    Where it runs
    the agent, churn-reason extraction, signal-kind labelling
    Why this one
    cheaper models fell below the accuracy bar
Chosen by bake-off against the accuracy bar, and swappable behind the one proxied endpoint.
The harness, in full

Every check, what it holds, and when it runs.

The ten mechanical checks behind section 05, as a table.

Where the system checks itself
  • Check
    Calibration gate
    What it holds
    A classifier may not run at full scale until it agrees with a human-labelled calibration set on at least 90% of rows. Results are cached, so a re-run never silently relabels. The gate is the rule, not a score — every gold row carries who reviewed it, edge cases are excluded from the primary metric, and the accuracy that matters is measured on the production run rather than on the set the prompt was written against.
    When
    before any full-scale run
  • Check
    Closed vocabularies
    What it holds
    Every label is a pick from a fixed list. The model cannot invent a category.
    When
    every row
  • Check
    Grounded answers
    What it holds
    The agent never recites a figure written into its instructions. Every number comes from a tool call against live data — including the coverage line, derived from MAX(period).
    When
    eval-enforced
  • Check
    Gold sets, never edited to fit
    What it holds
    The eval set is human-verified and fixed. When an answer disagrees with it, the answer is what changes. The labelling gold set is 40 hand-labelled rows, and the prompt was tuned on them — so it grades the model on cases it was built for, and real out-of-distribution accuracy runs a few points lower. Production accuracy is measured separately, by stratified spot-checks of 15 to 25 rows against the actual run.
    When
    every eval run
  • Check
    Live reference anchors
    What it holds
    Anchors recompute each run, so normal monthly data movement does not break the test set.
    When
    every eval run
  • Check
    Taxonomy drift monitor
    What it holds
    Watches the taxonomy for drift.
    When
    ongoing
  • Check
    Noise hidden, not deleted
    What it holds
    Filtered rows keep a reason flag and stay in the lake; noise is hidden at query time, so filter quality stays auditable.
    When
    at query time
  • Check
    Weekly judge
    What it holds
    Scores live answers for faithfulness, over-claiming, scope honesty and language match. A thumbs-up or thumbs-down from a user lands as a score on the exact trace it came from.
    When
    Mondays, 07:00 UTC
  • Check
    Deploy-gate eval
    What it holds
    Prompt and tool changes batch into one full run against the fixed gold set. A change counts as done only when the deployed system is confirmed to serve the new answer.
    When
    per deploy
  • Check
    Publish gate
    What it holds
    The month must be complete and every derived table fresh, or nothing publishes.
    When
    every monthly publish
Tests and the agent eval both run in CI before anything ships. A change is done when the deployed system is confirmed to serve the new answer, not when the branch is green.
Limits

What this system does not do.

Each one is a trade-off I chose, with its reason.

  • Labelling covers ICP sellers and roughly the last twelve months — a cut chosen on the measurement: with a 45-day half-life, that scope carries about 99.3% of the recency signal at roughly a quarter of the cost. Older and non-ICP voice is answered from a thinner pool, and the coverage fills in over time.
  • The no-login churn rule is switched off. The login feed lags real activity, so it is better to under-count churn than to mis-flag active customers. It comes back when a reliable activity feed exists.
  • need / impact is the softest of the six labels: it often needs product state the model cannot see, so it is used for coarse routing, never as a fine score.
  • The WhatsApp corpus will not grow. Live capture is off, and the corpus stays a fixed historical set.
  • Churn rate, ICP membership and the customer census are business-owned. The system uses them and never recomputes them.
The taxonomy and the domain-to-team map are employer-specific by design. The patterns are portable.