Peeking into the black box: Lessons from the front lines of machine-learning product launches

Grace Huang shares lessons learned from running and interpreting machine-learning experiments and outlines launch considerations that enable sustainable, long-term ecosystem health.

With advances in machine-learning algorithms and the democratization of big data technologies, machine-learning products have become ubiquitous—they are the de facto choice for powering the experiences that are now common place in many of our beloved apps and services, such as recommendation and personalization. While offline evaluations are routinely done to evaluate algorithm performance, results from live experiments more closely reflects real-world performance and is a necessary step toward a product launch in many companies. However, running experiments on machine-learning products poses unique challenges and requires consideration beyond what traditional experiments frameworks offer. Grace Huang shares lessons learned from running and interpreting machine-learning experiments and outlines launch considerations that enable sustainable, long-term ecosystem health.