TL;DR
Agent analytics measures how people use the AI agents in a product and whether the agents complete tasks, with funnels, retention, cohorts and experiments on agent events. It answers different questions from LLM observability (traces, tokens, latency) and evals (correctness on a test set): did the user get the task done, and did that change conversion and retention? Log sessions, steps, tool calls, task outcomes, handoffs and feedback as events in the warehouse, with the same user and account IDs as your product events, so they join to revenue and support data.
Agent analytics is the practice of measuring how people use the AI agents in a product, such as chatbots, copilots and autonomous agents, and whether those agents complete the tasks they are given, using the same funnel, retention and cohort methods product teams use for human users.
The term is also used in contact centers for the performance of human support agents; this guide is about AI agents. It covers what to measure, an event model for agent sessions, four example analyses, and why the data belongs in your data warehouse, next to the product and revenue data it has to be compared with.
Why agent analytics matters now?
AI agents now show up in SaaS products in two ways. Some are features: a support chatbot, a copilot that drafts a report, an agent that reconciles invoices. Others are users: a customer's AI assistant calling your API or MCP server on their behalf. Either way, part of the value your product delivers now passes through an agent, and the product team has to know whether that part works.
Most teams start with LLM observability tools. They record traces, token counts, latency and cost per call, which is what an engineer needs to debug a slow or failing chain. They do not tell you whether the user got what they came for, whether that user came back the next week, or whether accounts that rely on the agent renew. Those are product questions, and the methods for answering them already exist.
Agent analytics vs LLM observability vs evals
Three disciplines look at the same agent from different distances. Evals test it before release, observability watches each request, and agent analytics follows users and outcomes over weeks.
| Agent analytics | LLM observability | Evals | |
|---|---|---|---|
| Question answered | Do users get their task done, and does that change conversion and retention? | What happened inside this request, and why was it slow, costly or broken? | Does this prompt or model give correct answers on a fixed set of cases? |
| Data | Product events: sessions, tool calls, outcomes, handoffs and feedback, joined to users, accounts and revenue | Traces and spans, prompts and completions, token counts, latency, errors | Curated test cases with expected outputs, scored by rules, people or a model as judge |
| Owner | Product, growth and data teams | AI and platform engineers | AI and ML engineers |
| Tools category | Product analytics platforms, run on agent events | LLM observability and tracing platforms | Evaluation frameworks and test harnesses |
The three are complementary. Evals tell you a change is safe to ship, observability tells you why a request failed, and agent analytics tells you whether shipping the agent helped. Amplitude and Pendo have both launched agent analytics products inside their own platforms, which says something about where the category is heading: the methodology is product analytics. An agent session is a journey, a finished task is a conversion, and a user who comes back is retained.
What to measure: AI agent metrics?
Seven metrics cover most agents. Track each by task type, because a lookup and a multi-step workflow have very different baselines.
- Task completion rate. Sessions that end in a completed task, divided by sessions in which a task was attempted. Define completed from an observed outcome, such as the refund was issued, the report was exported or the ticket was closed, rather than from the agent's own message saying it is done.
- Steps to completion. The number of agent steps or user turns between the start of a session and the completed task. Report the median and the 90th percentile; a growing tail usually means the agent is looping or asking questions it does not need to.
- Fallback and handoff rate. The share of sessions that end with a handoff to a person, a support ticket, or the user switching to the manual workflow. Some handoffs are correct, such as a change that needs approval, so record the reason.
- Retries. How often a user rephrases or regenerates the same request, and how often the agent re-runs a failed tool call. User retries measure frustration; tool retries measure reliability and cost.
- Abandonment rate. Sessions that end with no completion, failure or handoff, usually defined by a period of inactivity such as 30 minutes. This is the silent failure: no error and no feedback, the user simply leaves.
- User feedback. Thumbs up or down, ratings and comments per response or per session. Report the positive rate and the coverage, the share of sessions with any feedback, because only a small and unrepresentative group of users leaves it.
- Downstream conversion and retention. Whether users or accounts that use the agent, or whose tasks complete, go on to convert, retain or expand at a different rate than comparable users who do not. This is the metric that says whether the agent is worth running.
An event model for agent sessions in the warehouse
Agent analytics needs events, not traces. Log one event per meaningful moment, with a shared session_id, the same user_id and account_id your product events use, and the version of the agent that ran. Raw prompts and completions can stay in a separate, access-controlled table; the analytics events carry metadata. (Modeling events in the data warehouse covers the table layouts.)
| Event | Fires when | Key properties |
|---|---|---|
agent_session_started | A user opens the agent or sends the first message | session_id, user_id, account_id, agent_name, agent_version, model, prompt_version, entry_point |
agent_step | The agent responds or takes an action | session_id, step_index, step_type, latency_ms, input_tokens, output_tokens |
tool_called | The agent calls a tool, an API or a retrieval step | session_id, tool_name, status, attempt, duration_ms, error_type |
task_completed | The intended outcome is observed | session_id, task_type, steps_count, duration_s |
task_failed | The agent gives up or the outcome check fails | session_id, task_type, failure_reason |
handoff_to_human | The session is routed to a person or to the manual flow | session_id, reason, ticket_id |
feedback_given | The user rates a response or the session | session_id, step_index, rating, has_comment |
Three design choices matter more than the event names. Put agent_version, model and prompt_version on the session-start event, so every later analysis can be split by them. Emit task_completed from the system that observes the outcome, not from the model's reply. And reuse your existing user and account identifiers, so agent sessions join to everything else. If outside AI agents use your product through an API or MCP server, log their calls with an actor_type property (human or agent) and the client name; the same model then describes agents as users.
With that in place, completion rate by agent version, counted per session, is an ordinary query. The join only counts completions inside the same session and within 30 minutes of its start, so a session that runs longer than that loses its completion; widen the window if your agent sessions do. The denominator here is every started session; filter it to sessions with an attempted task if you follow the definition above strictly (Postgres-style syntax):
SELECT
s.agent_version,
COUNT(DISTINCT s.session_id) AS sessions,
COUNT(DISTINCT c.session_id) AS completed,
COUNT(DISTINCT c.session_id) * 1.0
/ COUNT(DISTINCT s.session_id) AS completion_rate
FROM events.agent_session_started AS s
LEFT JOIN events.task_completed AS c
ON c.session_id = s.session_id
AND c.event_time <= s.event_time + INTERVAL '30 minutes'
WHERE s.event_time >= CURRENT_DATE - INTERVAL '28 days'
GROUP BY 1
ORDER BY 1;Four example agent analyses
A funnel from question to completed task
Build a funnel from agent_session_started to tool_called (for the tool the task needs, such as search_orders) to task_completed, with a 30-minute conversion window, broken down by task_type. The step with the biggest drop is where to look first. If most sessions reach the tool but few complete, the tool returns the wrong data or the agent misreads it. If few sessions reach the tool at all, the agent is not understanding the request. The guide to funnel analysis covers windows and step ordering in detail.
Retention of users who used the agent vs those who did not
Take the users who signed up in one month and split them into two cohorts: those with a task_completed in their first 14 days, and those without. Chart weekly retention on your core product action for eight weeks. A gap in favor of agent users is encouraging, but users who try a new agent are often more engaged already. Compare within similar groups, such as the same plan, signup week and activity level before first agent use, and treat the result as a lead. To prove the agent causes the difference, run an experiment.
Segmentation by agent version or model
Chart weekly completion rate, handoff rate and abandonment split by agent_version, then by model, task type, plan and language. This is where regressions show up. A release that scores better on evals can still lower completion in production, for example when a new prompt asks more clarifying questions and users leave before answering. If token counts are on agent_step, add cost per completed task to the same view.
An experiment comparing two prompts or models
Assign users, not sessions, to prompt A or prompt B with your experimentation tool or your own assignment code, so one person does not see both, and write each exposure to the warehouse. Use task completion rate as the primary metric and steps to completion as a secondary one. Add guardrails: handoff rate, support tickets per user, cost per session and four-week retention.
Reading the test on business metrics changes the decision. A cheaper model that completes slightly fewer tasks may be acceptable; one that completes the same tasks but doubles support tickets is not. That trade-off is only visible when the exposures are joined to support and billing data. Experiment analysis on the warehouse shows what such a result looks like.
Why agent analytics belongs in the data warehouse?
- Joins to the rest of the business. Revenue, plans, CRM records and support tickets are already in the warehouse. Whether accounts with high agent completion renew more often is one join, not an export project.
- One definition of a completed task. When the data team defines it once, the product team, the AI team and the leadership dashboard report the same completion rate instead of three.
- Nothing leaves the warehouse. Agent sessions often contain customer data. Keeping them under the warehouse's existing access controls is simpler than sending them to another vendor.
- Volume without per-event pricing. One agent session can produce dozens of events. In the warehouse you pay for storage and compute, not per event.
Agent analytics on your warehouse with Mitzu
Mitzu is an analytics platform on your data warehouse for BI, product analytics and experiment analysis, where the data team governs the metrics and everyone uses them through dashboards, an AI agent, Slack and their own AI assistants. It has no separate agent analytics module, LLM tracing or eval product. For the analyses above it does not need one: agent sessions, tool calls, task outcomes and feedback are events in your warehouse, and Mitzu runs funnels, retention, cohorts and segmentation on any events in your warehouse, so agent events work the same way as signups and purchases.
The data team can define a completed task once, for example as a dbt model of agent tasks with an outcome column and a completion-rate metric on top, and every dashboard and agent answer uses that definition. Tests of prompts or models are read from the exposures your assignment writes to the warehouse, with lift against control, a 95% interval, guardrails and health checks (experiment analysis).
Mitzu's own AI agent can answer a question such as why completion rate fell after the last agent release, breaking the change down by task type, model and plan. Monitoring agents can watch the agent KPIs: scheduled agents review a dashboard or KPIs on the days and times you pick and email the team on every run, or only when a condition you describe in plain English is met, with the analysis attached.
Start free on your warehouse, or book a demo to see your agent events in funnels and retention next to your product and revenue data.
FAQ
What is agent analytics?
Agent analytics is the practice of measuring how people use the AI agents in a product, such as chatbots, copilots and autonomous agents, and whether those agents complete their tasks. It applies product analytics methods, including funnels, retention, cohorts and experiments, to events from agent sessions, and connects the results to conversion, retention and revenue.
What is the difference between agent analytics and LLM observability?
LLM observability looks inside individual requests: traces, prompts, token counts, latency, cost and errors, so engineers can debug the system. Agent analytics looks at users and outcomes over time: whether tasks get completed, where sessions are abandoned or handed off, and whether people who use the agent convert and retain better. Most teams need both, and they are usually owned by different people.
What metrics should I track for an AI agent?
Start with task completion rate, steps to completion, fallback and handoff rate, retries, abandonment rate and user feedback, each split by task type and agent version. Then add the downstream metrics: conversion, retention and expansion of users or accounts that use the agent compared with similar ones that do not.
How do you measure whether an AI agent completed a task?
Record completion from an observed outcome, such as an order updated, a report exported or a ticket closed, rather than from the agent's own statement that it is done. Emit a task_completed event from the system that sees the outcome, with the session ID and task type, and compute completion rate as completed sessions over sessions where a task was attempted, within a fixed time window.
Do I need a dedicated tool for agent analytics?
Not necessarily. Amplitude and Pendo offer agent analytics products inside their platforms. If your agent events land in your data warehouse, any product analytics tool that runs funnels, retention and cohorts on warehouse data can analyze them, next to the revenue, CRM and support data already there.
