Ting-Fang Yen details a monitor for production machine learning systems that handle billions of requests daily. The approach discovers detection anomalies, such as spurious false positives, as well as gradual concept drifts when the model no longer captures the target concept. You'll see new tools for detecting undesirable model behaviors early in large-scale online ML systems.