A/B Testing Resources

Recommended Platforms

• Eppo

Frameworks

• Planout
• Wasabi

Articles

• Why overlapping confidence intervals don’t imply non-significance

Pinterest

• Benefits of hold-out groups
• Building Pinterest’s old A/B testing process
• Monitoring A/B experiments in real-time

Airbnb

• Experiments at Airbnb
• Selection Bias in Online Experimentation

Netflix

• Netflix experimentation platform
• Netflix experiment examples

Twitter

• Twitter’s experimentation platform
• Detecting and Avoiding Bucket Imbalance in A/B Tests
• Implications of use of multiple controls in an A/B test
• Power, Minimal Detectable Effect and Bucket Size Estimation in A/B Tests

Stripe

• Reproducible research at Stripe

Etsy

• Continuous experimentation at Etsy

Uber

• Building an Intelligent Experimentation Platform with Uber Engineering

LinkedIn

• XLNT Platform: Driving A/B Testing at LinkedIn

Stitch Fix

• So, You Need a Statistically Significant Sample?

Booking.com

• How Booking.com increases the power of online experiments with CUPED

Spotify

• Comparing quantiles at scale in online A/B-testing

Papers

Microsoft - almost all available from their excellent website

Best papers

• Online experimentation at Microsoft
• Controlled experiments on the web: survey and practical guide
• Seven rules of thumb for web site experimenters
• Online experiments: Practical lessons

Other good references

• Online experiments at large scale
• Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data
• Online controlled experiments (slides)
• Online controlled experiments: Lessons from Running A/B/n Tests for 12 years (slides)
• Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
• Trustworty Analysis of Online A/B Tests: Pitfalls, challenges and solutions
• Continuous Monitoring of A/B tests without pain: Optional stopping in Bayesian testing
• Measuring Metrics
• Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments
• Data-driven Metric Development for Online Controlled Experiments: Seven Lessons Learned
• What works in e-commerce - a meta-analysis of 6700 experiments

LinkedIn

• From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks

Google

• Measuring the user experience on a large scale
• Overlapping Experiment Infrastructure: More, Better, Faster experimentation

Netflix

• Improving the Sensitivity of Online Controlled Experiments
• The Netflix Recommender System: Algorithms, Business Value, and Innovation

Facebook

• Uncertainty in online experiments with dependent data: An evaluation of bootstrap methods

DoorDash

• Meet Dash-AB — The Statistics Engine of Experimentation at DoorDash

Other

• A/B Testing using the negative binomial distribution in an internet search application
• Confidence intervals for AB-test
• Econometrics of Randomized Experiments
• A/B testing with limited data
• A/B testing using the Beta distribution

Good ideas for A/B test experiments from the papers

• Combined experiments. Per-user with x% chance of seeing treatment
• Metrics definitions as discussed in the Microsoft paper
• Learning experiments where you intentionally degrade the experience to see how it affects the baseline
• Digging deeper into adoption and retention
• Surprising results should be replicated
• Risk of focusing on small changes is incrementalism. Should be tried to get some high ROI but also some big bets for audacious goals
• Changes rarely have a big impact to key metrics – corollary: only ~10% of experiments have any positive result
• Metrics improvements should be diluted to their segment size (mobile -> overall MW)
• Borderline significant results should be tentative and experiments rerun to verify
• If the result is so good (e.g. 8 sd from mean) then check again even if statistically significant
• Best to test yourself as many explanations for amazing A/B test results are wrong
• Can quantify latency’s effects by artificially slowing down a site
• Can we do a speed-up experiment by allowing response and delaying loading social column?
• Reducing abandonment is hard, shifting clicks is easy
• Delta method instead of bootstrap when data sizes are large (See Casella & Berger - Statistical Inference or Wasserman - All of Statistics)
• ANOVA as an alternative to t-tests when comparing the means of more than 2 samples
• Check experiment groups religiously for equal sizes and variances via A/A tests
• Check for browser-specific bugs
• Filter out users who didn’t even reach the page you have the treatment on for lower variance and better power (aka triggering)
• Check for server-related caching issues