How to Analyze Optimizely, VWO and LaunchDarkly A/B Tests on Your Data Warehouse

To analyze A/B tests from Optimizely, VWO or LaunchDarkly on your own warehouse, land the tool's exposure events in the warehouse and join them to the metric tables your business already reports on, counting only what each user did after their first exposure. Then compute the lift and a confidence interval for each metric, run health checks before you read the result, and keep the experimentation tool for what it does well: assigning users and delivering variants.

This guide walks through that process on Snowflake, BigQuery, Databricks or Redshift: the data you need, the SQL for a conversion metric, the statistics per metric type, the checks that catch broken tests, and how to read the outcome. It assumes your event and order data is already in a warehouse.

Why analyze experiments on the warehouse?

Experimentation tools report results on the events they collect and the metrics you configure inside them. That works until someone asks how the test affected the number in the weekly business review, and the two do not match. Reading results on the warehouse fixes four things.

• Results on the metrics the business reports. Net revenue, orders and active users come from the same tables and definitions as the KPI dashboard, so a 3% lift in revenue is a lift in the revenue finance counts.
• Joins the testing tool cannot make. Refunds from the payments system, cancellations from billing, tickets from support and retention from product events all sit next to the exposures. A checkout change that lifts orders and also lifts refunds shows up as both.
• One source of truth. Product, data and finance read the same result instead of reconciling a vendor dashboard with an internal one.
• Long-term effects. The exposure table stays in your warehouse after the test ends, so you can check 90-day retention or a second purchase months later, on the same users.

What data you need?

Two kinds of table: an exposure table that records who saw which variant and when, and the metric tables you already have. The exposure table needs four columns.

Optional columns such as platform, country or app version help with segment breakdowns later. The metric side is whatever already exists: an orders table, a subscriptions table, product events. If your exposures arrive as a nested array or JSON property on every event rather than a separate table, modeling A/B tests in the data warehouse covers how to flatten them.

Getting exposures out of Optimizely, VWO or LaunchDarkly

There are two general routes, and they work the same way for any experimentation or feature-flag tool:

• The vendor's data export or event stream. Most experimentation tools can deliver their exposure or decision events to cloud storage or straight into a warehouse. Check your plan and the vendor's documentation for what is available and how often it lands.
• Log your own exposure event. When your code asks the tool which variant to show, send an exposure event through your existing event pipeline, for example a CDP such as Segment or RudderStack, or your own tracking. This route works with any tool and with in-house assignment.

Two details decide whether the data is usable. First, log the exposure when the user actually reaches the changed experience, not when they are assigned on page load somewhere else; users who never see the change only dilute the effect. Second, make sure the ID is the one your metric tables use. If the tool randomizes on an anonymous device ID and orders are keyed by a logged-in customer ID, you need an identity mapping before any join will work.

How to analyze an A/B test on your warehouse, step by step?

1. Reduce exposures to one row per user

A user who loads the checkout five times produces five exposure rows. For the analysis you want one row per user: the variant and the timestamp of the first exposure. Count the distinct variants per user at the same time, because you will need that number for a health check.

2. Define each metric once

Use the definitions the business already agreed on rather than writing new ones per test. Most experiment metrics fall into three types, and the type decides the statistics later:

• Proportion: the share of exposed users who did something, such as placed an order or started a trial.
• Mean: a per-user average, such as revenue per exposed user or sessions per user.
• Ratio: one sum divided by another where the denominator is not the user count, such as average order value (revenue / orders) or refund rate (refunded orders / orders).

3. Attribute outcomes to users after their first exposure

Count only what happened after a user's first exposure, inside a fixed window such as 14 days. Anything before exposure cannot have been caused by the variant. A fixed window also gives every user the same amount of time to convert, so a user exposed on day one does not count for more than a user exposed on the last day. And keep every exposed user in the denominator, including those with no outcome at all.

The query below does this for order conversion, in Snowflake syntax:

WITH first_exposure AS (
  SELECT
    user_id,
    MIN(variant)            AS variant,
    MIN(exposed_at)         AS first_exposed_at,
    COUNT(DISTINCT variant) AS variant_count
  FROM analytics.experiment_exposures
  WHERE experiment_key = 'one_page_checkout'
    AND exposed_at >= '2026-09-01'
    AND exposed_at <  '2026-09-29'
  GROUP BY user_id
),

exposed_users AS (
  SELECT user_id, variant, first_exposed_at
  FROM first_exposure
  WHERE variant_count = 1
),

converted_users AS (
  SELECT DISTINCT e.user_id
  FROM exposed_users AS e
  JOIN analytics.orders AS o
    ON  o.user_id = e.user_id
    AND o.ordered_at >= e.first_exposed_at
    AND o.ordered_at <  DATEADD(day, 14, e.first_exposed_at)
  WHERE o.status = 'completed'
)

SELECT
  e.variant,
  COUNT(*)                          AS exposed_users,
  COUNT(c.user_id)                  AS converted_users,
  COUNT(c.user_id) / COUNT(*)::FLOAT AS conversion_rate
FROM exposed_users AS e
LEFT JOIN converted_users AS c
  ON c.user_id = e.user_id
GROUP BY e.variant
ORDER BY e.variant;

Three things in that query carry the method. The variant_count = 1 filter drops users who were exposed to both variants (more on that below). The join condition limits orders to the 14 days after each user's own first exposure. The final LEFT JOIN keeps non-converting users in exposed_users; an inner join would silently compute conversion among buyers only. In BigQuery, replace DATEADD with TIMESTAMP_ADD(e.first_exposed_at, INTERVAL 14 DAY) and the cast with SAFE_DIVIDE. Exposures stop on September 28, so run the final analysis on or after October 13, when the last users have had their full 14 days.

4. Compute the lift and its interval for every metric

Suppose the query returns 1,200 conversions from 20,000 control users (6.0%) and 1,320 from 20,000 treatment users (6.6%). The relative lift is 10%. The 95% interval on the difference is 0.12 to 1.08 percentage points, and a two-proportion z-test gives z = 2.47 and p = 0.014. The interval excludes zero, so the difference is significant at the 5% level, and it also tells you the plausible size of the effect, which the p-value alone does not.

Each metric type needs its own variance. Proportions use the binomial variance in a z-test. Means use a t-test on per-user values, computed over all exposed users with zeros for those who did nothing. Ratio metrics are the trap: orders are not independent of each other when one user places several, so a naive t-test on order values understates the variance. Use the delta method or a bootstrap over users. A/B testing statistics for beginners explains significance, intervals and power in more detail.

5. Run health checks before you read the result

A significant result from a broken test is still a broken result. Check three things first:

• Sample ratio mismatch (SRM). Compare the users per variant with the planned split using a chi-square test. A planned 50/50 test with 20,000 and 19,200 users gives a chi-square of 16.3 and p < 0.0001: something in assignment or logging is dropping users from one side, and the lift cannot be trusted until you find it. Sample ratio mismatch has the SQL for this check.
• Users in more than one variant. Count users with variant_count > 1. A handful is normal (shared devices, ID merges). A large share, or a share that differs between variants, points to an identity or caching bug, and excluding those users can bias the result.
• Window coverage. Check that every included user has had the full attribution window. If you analyze on day 20 of a test with a 14-day window, users exposed in the last two weeks are only partly observed, and early exposures dominate the result.

6. Correct for multiple comparisons

Reading ten independent metrics at a 5% threshold gives a 40% chance that at least one comes out significant by luck when nothing changed. With twenty, it is 64%. Name one primary metric before the test starts, treat the rest as secondary or as guardrail metrics, and apply a correction across them: Bonferroni and Holm control the chance of any false positive, Benjamini-Hochberg controls the share of false positives among the metrics you call significant.

7. Read the result

• Decide on the primary metric's interval. If its lower bound is above zero and the effect is large enough to matter, the variant won. If the interval spans zero, the test is inconclusive at this sample size, which is not the same as showing there is no effect.
• Check the guardrails. A checkout that lifts conversion and also lifts refund rate or support tickets may not be a win once both are counted.
• Analyze at the planned end. These are fixed-horizon tests. Checking every day and stopping at the first significant p-value inflates false positives; run to the sample size you planned.
• Look at segments last. Breakdowns by platform or country are useful for understanding a result, and they are more comparisons, so treat a surprising segment as a hypothesis for the next test.

Doing it in Mitzu

Mitzu is an analytics platform on your data warehouse for BI, product analytics and experiment analysis. For experiments, it reads A/B test exposures from your warehouse and reports lift, confidence intervals, guardrails and health checks on the same metrics your business reports; your experimentation tool keeps assigning users.

• Setup: it works with any tool that records exposures in your warehouse, such as Optimizely, VWO or LaunchDarkly, or your own in-house assignment. Mitzu needs the exposure event and the property that holds the variant.
• Statistics: frequentist tests. Every metric gets its lift against control, a 95% interval and a p-value, two-sided by default, with a correction for multiple comparisons. Proportion, mean and ratio metrics are supported.
• Health checks: whether traffic is split as planned, whether users appear in several variants, how much of the test window the results cover, and the sample against plan, on every experiment.
• Same definitions: results use the same metric definitions as your dashboards and product analytics, so an experiment's revenue is the revenue the business reports.
• The agent: ask which variant won, why a guardrail moved, or how a segment responded, and the analytics agent answers on the same experiment data and definitions.

See the experiment analysis page for screenshots of a result and its health checks, start free on your own warehouse, or book a demo and we will walk through one of your tests on your data.

FAQ

Do I need to replace my experimentation tool?

No. Optimizely, VWO, LaunchDarkly or your in-house system keeps assigning users, delivering variants and managing rollouts. Analyzing on the warehouse only changes where you read the results: on the exposures the tool records, joined to your own metric tables.

Why do my warehouse results differ from my experimentation tool's results?

Usually because the inputs differ. The tool may count conversions from its own client-side events while the warehouse counts completed orders net of cancellations. Attribution windows, bot filtering, identity resolution and what counts as an exposure can all differ too. Line them up one at a time, starting with exposed users per variant.

How long should the attribution window be?

Long enough for the behavior the metric measures to happen: a few days for a checkout conversion, 14 or 30 days for a trial-to-paid conversion, longer for retention. Pick it before the test starts and use the same window for every user.

Can I analyze A/B tests in Snowflake or BigQuery with SQL alone?

Yes for proportions and means: the counts, sums and sums of squares come from SQL, and the z-test or t-test is a few lines of arithmetic on top. Ratio metrics and multiple-comparison corrections take more work, and the hard part over time is keeping the method identical across every test and every analyst.

What is warehouse-native experimentation?

Analyzing A/B tests on the exposure and event data already in your warehouse, rather than in a separate tool's copy of it. The results use the same metric definitions as your dashboards and product analytics, and the data never leaves the warehouse.