Building an AI data analyst the whole company trusts
Anyone at Dutch asks a data question in Slack and gets a governed answer in seconds. Making those answers trustworthy enough for the whole company to run on took real work, and it is what this case study covers.
From a growing queue to answers the whole company trusts
The queue
Every data question funneled to a small BI team.
A wrong answer
The bot answered confidently, and was quietly wrong.
Controls
Each failure became a permanent, automated control.
Evals
A nightly suite grades every answer against truth.
Trust
The whole company asks directly, and relies on it.
What talking to Dash looks like
The answer up top is the simple case. Two harder ones matter more: following up in context, and what happens when the bot doesn't know.
Sanitized exchanges; figures are illustrative.
Centralized BI, growing queue
Dutch runs centralized BI on purpose. End users don't write SQL, and my team governs every data model, so definitions stay consistent across the company. The tradeoff is a queue, because every question about revenue, consults, refunds or retention landed on a small team, and the queue grew faster than we did.
An AI analyst was the obvious answer and the obvious risk. A wrong dashboard looks broken, but a wrong AI answer looks exactly like a right one, delivered with confidence to someone who has no way to check it.
Everything lives in the warehouse
The Slack bot is a thin webhook with no intelligence of its own. The answering logic sits inside Snowflake, next to the data, in two governed layers built with dbt.
a person asks a question in plain English
routes to 1 of 8 semantic views, one per business domain
synonyms, metric definitions and verified queries the AI must use
~70 dbt models, one recipe per metric, shared with every dashboard
Every failure became a guardrail
Early on I spot-checked one answer against finance. It came back confidently, plausibly wrong.
"I only caught it because I happened to look. Nothing in our process would have caught it otherwise."
The system got sturdier every month. Each failure mode surfaced once, then closed for good.
Evals are the product
Underneath all of it is a three-tier evaluation stack, every check graded against truth SQL written independently of the AI layer. The eval code gets the same review and fix discipline as anything else in production, because it needed it. The grader once failed a correct answer over a "$3.14M" it read as 3.14.
sv-drift-audit.yml · on: scheduleGolden questions
Asked in natural language, graded against independent truth SQL within tight tolerances. Failures page Slack before the workday starts.
Paraphrases
The same metrics in alternate wordings real users typed. The model is stochastic, so a wording fails only on two consecutive misses. A failing wording becomes a verified query, permanently pinned.
Sampled real traffic
A standing audit pulls real questions from the usage log each month and verifies the answers. The nightly and weekly checks use questions we wrote. This one uses the questions people actually asked. Weak spots turn into new verified queries.
Built to be trusted, not just fast
On top of the guardrails from those incidents and the nightly evals, a fixed set of protections ships with every answer.
PII guardrail
Never returns names, emails or anything that identifies a person.
Cited sources
Every answer names the table or view the number came from.
Freshness
Answers about today carry an "as of" timestamp and a note on the pipeline.
Accuracy alerts
When someone flags a wrong answer, the data team is notified in #dash-data-alert automatically.
Query timeout
A hard timeout stops a runaway query from tying up the warehouse.
Stays in its lane
Off-topic questions get a polite redirect instead of a guess.
Every change ships as a pull request
The AI layer is configuration, and configuration drifts when humans paste it into consoles. We moved the semantic views into dbt, wired a deploy-on-merge workflow, and retired the console path. Git equals live by construction, and review happens on a pull request like any other code change.
The new path proved itself on day one by refusing to deploy over a permission gap a manual deploy would have hidden for weeks.
Before
edit SQL in a console editor
paste, save, hope git stays current
drift audited after the fact, nightly
After
open a pull request
merge = deploy, via CI as a service role
drift structurally impossible
New questions, same spine
Once the governance held, we could add new kinds of questions without loosening any of it. Each new ability rides the same layers, tests and deploy path.
Dashboard discovery
People kept asking which dashboard a number lives on. The bot now searches across 110 production dashboards and returns the best matches, each with a one-line note on what it covers and a link to open it.
Content engagement
We pulled product-analytics events into the governed layer, so the bot answers how content performs by audience. It is tested and deployed exactly like every other metric.
What changed
- Marketing, operations, CX and the executive team ask the bot directly. Most of those questions never reach the BI queue.
- A user asked for a metric the bot didn't have; the governed view, tests, AI exposure and a verified query shipped in one day.
- Every wrong-answer class we have hit now has a named, automated control.
- Semantic view changes ship through pull requests with automated deploys, and drift between definition and production is structurally impossible.
- The bot's reach grew past finance metrics into dashboard discovery and content engagement, and each new question type goes through the checks already in place.
What I'd tell another data leader
- An AI analyst without evals is a liability with a chat interface.The demo took a week. The trust took months, and most of it was spent fixing our own mistakes one at a time.
- Keep tolerances tight and fix the window definitions instead of loosening the check.Loosening a tolerance would have hidden the miss instead of fixing it.
- Put the AI layer in git and deploy it like code.It drifts the way code does, so it goes through review and CI like code.
- Write your truth queries independently of the system you are grading.Otherwise you are just checking that the system agrees with itself.