Marketplace · Product
Search ranking A/B test
Reading a search ranking experiment on your warehouse, with new-seller share and cancellations as guardrails.
New ranking vs. current ranking
+8.9%
Significant winSearch-to-booking rate
095% interval +4.1% to +13.7%
New ranking
Current ranking
control
Who it's for
Product and search teams who test ranking changes and need to see the effect on both buyers and sellers.
What it solves
Analyze the ranking test on the exposure data in your warehouse, with the same booking definition as your dashboards, and check that new sellers do not lose out.
Questions it answers
Did the new ranking lift bookings per search?
Search-to-booking rose from 24.6% to 26.8%, an 8.9% lift (95% interval +4.1% to +13.7%).
Did new sellers lose visibility?
New sellers kept an 11% share of bookings, within the guardrail, and cancellations did not change.
- Is the lift significant, and how wide is the interval?
- Which categories and cities responded best?
The impact
- Ship ranking changes on evidence from both sides of the marketplace
- One booking definition for experiments and reporting
- Catch tests that help buyers but starve new sellers
The data it needs
- Exposure events from your experimentation or feature-flag tool
- Search, listing view and booking events
- Seller attributes, such as tenure and category
Read in place in your warehouse. Nothing is copied out.
Search ranking A/B test: frequently asked questions
Run this on your own data.
Start free on your warehouse, or book a demo and we'll show it on your data.