A/B testing at scale: Accelerating software innovation

Controlled experiments such as A/B tests have revolutionized the way software is being developed, allowing real users to objectively evaluate new ideas. Ronny Kohavi, Alex Deng, Somit Gupta, and Paul Raff lead an introduction to A/B testing and share lessons learned from one of the largest A/B testing platforms on the planet, running at Microsoft, which executes over 10K experiments a year.

The internet provides developers of connected software, such as websites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments, also known as A/B tests. Search engines, retailers, social networking services, travel services, and startups of all stripes now use online controlled experiments to make data-driven decisions at a wide range of companies in everything from frontend user-interface changes to backend algorithms. The theory of a controlled experiment is simple and dates back to Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s. However, the deployment and evaluation of online controlled experiments at scale (think hundreds of concurrently running experiments) across variety of websites, mobile apps, and desktop applications presents many pitfalls and new research challenges. Ronny Kohavi, Alex Deng, Somit Gupta, and Paul Raff lead an introduction to A/B testing and share lessons learned from one of the largest A/B testing platforms on the planet, running at Microsoft, which executes over 10K experiments a year. You’ll discover practical and research challenges in scaling experimentation and promising directions for future work.