Why an Analytics Agent Needs a Semantic Layer

Point an LLM at a thousand-table warehouse and ask about activation. It will find something — a table that looks like events, a column that looks like a user id, a status field whose values it guesses. The query runs, a number comes back, and nobody can say where it came from. This is the trust problem every analytics agent has to solve, and the solution is not a better model. It is better context.

That context layer is the semantic layer: the agent’s grounded vocabulary of what exists in the data and what it means. This post covers what a semantic layer has to contain for an analytics agent to be trustworthy — and why the BI-shaped version most tools use isn’t enough for product analytics.

Three ways an ungrounded agent fails

• Invented vocabulary. Without a catalog of real events and properties, the agent guesses names — filtering on country = 'Germany' when the column holds DE, or querying a signup event that is actually called account_created. The query succeeds and silently returns the wrong population.
• Wrong methodology. Funnels need conversion windows and step ordering; retention needs cohort bucketing; segmentations need user deduplication. An agent improvising SQL gets these subtly wrong in ways that read as plausible numbers.
• Unverifiable answers. If every answer is produced by a fresh, freehand generation, an analyst has to audit each query individually. That doesn’t scale past a demo, and it is why many agent rollouts stall after the first wrong number reaches a decision.

The accuracy gap, measured

These failure modes are not hypothetical edge cases. Snowflake's own engineering benchmark measured a 51% accuracy baseline for single-shot GPT-4o text-to-SQL on real-world BI prompts (Snowflake Engineering) — a coin flip is not production-safe for decisions. Independent hallucination benchmarks show frontier models hallucinating on roughly 60% of hard multi-turn tasks without external grounding (HalluHard, 2026).

The numbers make the design constraint concrete: an agent whose queries come from freehand generation will be wrong often enough that every answer needs auditing — which defeats the point of having an agent. Grounding and determinism are how the audit burden goes away.

What a product-analytics semantic layer stores?

Most agentic analytics tools ground the AI in a BI semantic layer: metrics, dimensions, entities, joins, hand-authored in YAML (dbt MetricFlow, Cube, LookML). That shape was designed for dashboards, and it has two problems as agent context: it takes weeks of data engineering to write and maintain, and it cannot express the concepts product analytics runs on.

A semantic layer specialised for product analytics stores a different set of things:

• Events and event properties — the behavioural vocabulary: what users do, with which attributes
• Entities — users, sessions, accounts, teams — and the dimension properties attached to them
• Sampled property values — real values pulled from the warehouse, so filters get suggested from what actually exists rather than invented by the model
• Join relationships — how events connect to users, accounts, and the dimension tables around them
• Accumulated context — saved insights, dashboards, and named cohorts that become vocabulary for future questions

Sampled values deserve emphasis, because they close the most embarrassing failure mode. When the semantic layer knows the real values of plan_tier or country, the agent filters on values that exist. Nothing is left to the model’s imagination.

Built automatically, maintained where people can see it

A semantic layer that takes weeks of YAML authoring doesn’t just delay setup — it goes stale, because updating it competes with everything else on the data team’s backlog. An agent grounded in a stale layer confidently answers from last quarter’s schema.

Mitzu’s answer is to make construction itself agentic. The Configuration Agent scans your data warehouse, identifies event and dimension tables, recognises common schemas (Segment, Snowplow, Firebase, GA4, custom event tables), maps user and group identifiers, and configures join relationships. The analyst reviews and adjusts — nobody hand-writes YAML. The layer lives in the app where analysts, PMs, and marketers can all read it, and the agent helps keep it current as new data lands.

The semantic layer alone isn’t enough: methodology and determinism

Grounding fixes vocabulary. It does not fix methodology. Even with perfect context, an LLM authoring funnel SQL can still botch the conversion window — the semantic layer told it what the events are, not how a funnel must be computed. This is the central argument of analytics agents vs text-to-SQL and BI chatbots: methodology has to live in code, not in a prompt.

In Mitzu’s architecture the two layers work together. The semantic layer gives the Analytics Agent its vocabulary; the agent assembles an analysis specification — funnel steps, conversion window, breakdown; or cohort definition, return event, time granularity — and a deterministic query engine turns that specification into SQL. The agent never writes SQL. The same specification produces the same SQL and the same answer, every time.

What correct funnel SQL has to enforce?

To make "methodology" concrete, here is what a correct three-step funnel has to enforce at the SQL level: first qualifying event per user per step, strict step progression, and an explicit conversion window. The annotations mark the rules a freehand generation most often drops.

-- Funnel: signup -> connect_data_source -> first_report within 14 days
-- Method rule 1: first occurrence per user per step
WITH first_signup AS (
  SELECT
    user_id,
    MIN(event_time) AS signup_time
  FROM analytics.events
  WHERE event_name = 'signup_completed'
  GROUP BY 1
),
first_connect AS (
  SELECT
    e.user_id,
    MIN(e.event_time) AS connect_time
  FROM analytics.events e
  JOIN first_signup s ON s.user_id = e.user_id
  WHERE e.event_name = 'data_source_connected'
    AND e.event_time >= s.signup_time -- rule 2: enforce order
    AND e.event_time < TIMESTAMP_ADD(s.signup_time, INTERVAL 14 DAY) -- rule 3: window
  GROUP BY 1
),
first_report AS (
  SELECT
    e.user_id,
    MIN(e.event_time) AS report_time
  FROM analytics.events e
  JOIN first_connect c ON c.user_id = e.user_id
  WHERE e.event_name = 'report_viewed'
    AND e.event_time >= c.connect_time -- rule 4: strict progression
    AND e.event_time < TIMESTAMP_ADD(c.connect_time, INTERVAL 14 DAY)
  GROUP BY 1
)
SELECT
  COUNT(DISTINCT s.user_id) AS step_1_signup_users,
  COUNT(DISTINCT c.user_id) AS step_2_connected_users,
  COUNT(DISTINCT r.user_id) AS step_3_report_users,
  SAFE_DIVIDE(COUNT(DISTINCT r.user_id), COUNT(DISTINCT s.user_id)) AS end_to_end_conversion_rate
FROM first_signup s
LEFT JOIN first_connect c ON c.user_id = s.user_id
LEFT JOIN first_report r ON r.user_id = s.user_id;

Every one of those rules is a place a freehand generation can silently go wrong — one global window instead of per-step windows, order not enforced, users counted across incompatible paths. In Mitzu these rules are not re-derived per question: they live in the deterministic engine, and the agent's specification only chooses the events, window, and breakdown.

SQL transparency: trust needs an audit trail

Every answer in this architecture comes with the SQL the engine generated, open for inspection. Note what this changes about the review burden: the analyst is verifying the output of a deterministic engine — the same code path every funnel goes through — rather than auditing a fresh LLM generation per question. Verify the methodology once, and it holds for every answer the engine produces.

Precision matters when vendors say “hallucination-free.” In Mitzu, the SQL is deterministic — the engine cannot hallucinate a query. The agent’s natural-language summary is still model-generated prose, which is exactly why the SQL and the underlying charts stay one click away.

This combination — grounded vocabulary, deterministic SQL, reviewable output — is what lets a data team put an agent in front of PMs and marketers without becoming the fact-checking department. It is also what a warehouse-native architecture protects: the layer describes data that stays in your warehouse, a point covered in vendor-silo analytics agents vs warehouse-native.

An evaluation checklist for data teams

Whatever tool you evaluate, the same tests apply. An agent rollout survives contact with real stakeholders when:

• Metric and event definitions live in one governed semantic source the agent actually reads.
• Funnel, retention, and segmentation methodology is enforced by the system, not left to per-question prompting.
• Every AI-produced answer exposes its SQL for review.
• Execution is warehouse-native, so answers reflect live governed data rather than stale copies.
• A sample of agent answers gets audited for method compliance during rollout.

To go deeper on the layer itself, the Mitzu semantic layer page covers how definitions are stored and governed, and the AI analytics agents for your warehouse page shows how the Configuration Agent and Analytics Agent work together from setup to answer.

FAQ

What does a semantic layer do for an analytics agent?

It gives the agent a grounded vocabulary: which events, properties, and entities exist, how they join, and what real property values look like. Grounded in that catalog, the agent builds analyses from things that exist instead of guessing table and column semantics.

How is a product-analytics semantic layer different from a BI one?

BI semantic layers (dbt MetricFlow, Cube, LookML) define metrics, dimensions, and joins — a shape designed for dashboards, authored by hand in YAML. A product-analytics semantic layer stores events, entities, dimension properties, and sampled property values, and pairs with an engine that knows funnel, retention, and cohort methodology.

Does a semantic layer prevent AI hallucinations?

It prevents a specific class of them: invented events, properties, and filter values. Methodology errors need a second mechanism — a deterministic query engine so the SQL is generated by code rather than by the model. In Mitzu, the SQL output is deterministic; the agent’s prose summary is still worth reading against the charts.

Who maintains the semantic layer?

In Mitzu, the Configuration Agent builds it by scanning the warehouse, and analysts review and adjust in the app — no YAML files to version and maintain. Saved insights, dashboards, and named cohorts extend it automatically as the workspace gets used.