TL;DR
Text-to-SQL agents and BI chatbots generate queries; analytics agents run investigations. See why generic SQL generation falls short for product analytics.
Every analytics vendor now has a chat box. Type a question, get a chart. But the architecture behind that chat box varies enormously, and the differences only show up after the demo — on the questions your team actually asks. The most common architecture is text-to-SQL: an LLM writes a query against your warehouse, sometimes grounded in a semantic layer of metrics and dimensions.
Text-to-SQL is genuinely useful for a class of questions. It is also structurally unsuited to product analytics. This post explains where the line sits, and what an analytics agent does differently.
What text-to-SQL and BI chatbots do?
Text-to-SQL tools — Snowflake Cortex Analyst, Databricks Genie, Looker's agent, ThoughtSpot Sage, or ChatGPT pointed at a warehouse — share one architecture: the AI generates SQL or a query DSL, executes it, and presents the result. A semantic layer of metrics, dimensions, and joins can ground the generation, which meaningfully improves accuracy on metric lookups.
On descriptive questions this works well. What was MRR last quarter? How many weekly actives in Germany? Revenue by plan, last six months? These map onto a single SELECT over well-modelled tables, and a grounded LLM produces that query reliably. Data analysts and data scientists — the primary users of these tools — can also read the output and catch mistakes.
Where generic SQL generation breaks: methodology?
Product analytics questions are different in kind, not just in difficulty. A funnel is not a SELECT — it is a sequence of per-user event matches with strict ordering and a conversion window. Retention requires cohort assignment, time bucketing, and a return-event definition. A journey requires per-user path reconstruction. These are methods, not metrics.
When an LLM authors this SQL directly, the failure mode is not a syntax error — the query runs and returns rows. The failure mode is a methodology error that produces a plausible, wrong number:
- A funnel without a conversion window, or with steps counted out of order
- Retention that doesn’t enforce cohort time bucketing, so cohorts bleed into each other
- A segmentation that double-counts users who triggered an event multiple times
- Filter values the model invented because it never saw the real ones in the data
These errors are hard to catch precisely because the output looks right. A PM who receives a conversion rate of 4.2% has no way to know the window was wrong. An analyst can audit one query — but not every query an agent runs across dozens of tool calls.
Why BI semantic layers don’t close the gap?
The standard answer to LLM query errors is a semantic layer: define metrics and dimensions once, and let the AI query the definitions instead of raw tables. That helps — for BI-shaped questions. But semantic layers in this space (dbt MetricFlow, Cube, LookML) express metrics, dimensions, entities, and joins. They have no native vocabulary for funnels, retention windows, cohort definitions, or journey depth.
So even a well-grounded text-to-SQL agent falls back to freehand SQL exactly where the methodology risk is highest. And building that semantic layer is data engineering work — typically weeks of YAML authoring before the first question gets answered. The role a semantic layer should play for an agent is covered in why an analytics agent needs a product-analytics semantic layer.
What an analytics agent does instead?
In agentic product analytics, the agent doesn’t write SQL at all. It assembles an analysis specification — for a funnel: first event, subsequent events, conversion window, breakdown; for retention: cohort definition, return event, time granularity. A deterministic query engine turns that specification into SQL using product analytics methodology encoded in code, not in a prompt.
The division of labour is the point. The LLM does what it is good at — interpreting the question, planning the investigation, deciding which analyses to run. The engine does what it is good at — generating correct, consistent SQL. Same specification, same SQL, same answer, every time.
The SQL is still shown for review, but it is a verification artifact, not the model’s authored work.
| Question | Text-to-SQL / BI chatbot | Analytics agent (agentic product analytics) |
|---|---|---|
| What was MRR this quarter? | Works well — single metric query | Works — descriptive queries are the easy case |
| Signup-to-activation funnel by channel, 7-day window | Methodology risk: windows and step order left to the LLM | Funnel methodology enforced by the query engine |
| Why did week-2 retention drop in November? | Returns a chart, not an investigation | Agent fans out into breakdowns and cohort comparisons, then synthesises |
| Show users in DACH who completed checkout this month | May invent region codes | Filters suggested from sampled values in the warehouse |
| Fit a model to predict 30-day churn | Not the tool | Not the tool either — that’s notebook work |
When text-to-SQL is the right choice?
This is not an argument that text-to-SQL is bad technology. If your questions are descriptive and BI-shaped — financial metrics, operational reporting, arbitrary one-off queries over modelled tables — a grounded text-to-SQL agent in the hands of an analyst is a real productivity gain. The same goes for notebook AI features used by data scientists for bespoke statistical work.
The mismatch appears when the questions are behavioural — funnels, retention, journeys, cohort comparisons — and the audience includes people who cannot audit SQL. That combination is exactly what product analytics is, and it is where the methodology needs to live in an engine rather than in a model’s output.
Where Mitzu sits?
Mitzu is an agentic product analytics platform that runs on your data warehouse. The Analytics Agent answers behavioural questions by assembling analysis specifications that a deterministic query engine turns into SQL — the same engine that has generated SQL for Mitzu’s UI for years. The semantic layer underneath is specialised for product analytics and built automatically by scanning the warehouse.
If your event data already lands in Snowflake, BigQuery, Databricks, Redshift, or ClickHouse, you can see how this architecture behaves on your own questions — the AI analytics agents on your data warehouse page walks through the agent surfaces and capabilities. For the data-residency side of the comparison, see vendor-silo analytics agents vs warehouse-native.
FAQ
Is a BI chatbot the same as an analytics agent?
No. A BI chatbot translates one question into one query. An analytics agent plans and runs a multi-step investigation, then synthesises an answer. Many products labelled “agents” are architecturally chatbots — the test is whether a diagnostic question triggers an investigation or just a chart.
Does a semantic layer fix text-to-SQL accuracy?
It improves accuracy on metric and dimension lookups, which is why every serious text-to-SQL tool uses one. It does not fix product analytics methodology, because BI semantic layers cannot express funnels, retention windows, or cohort logic — the LLM still authors that SQL freehand.
Why does deterministic SQL generation matter?
Because it makes answers reproducible and auditable. When a deterministic engine generates SQL from an analysis specification, the same question yields the same SQL and the same number every time — and analysts can review the engine’s output once instead of auditing every individual LLM generation.
Who should still use text-to-SQL tools?
Data analysts and data scientists running descriptive, BI-shaped, or one-off exploratory queries — especially where they can read and correct the generated SQL. For behavioural product questions consumed by non-technical teams, an analytics agent with methodology in the engine is the safer architecture.
