How to make analytic operations look more like DevOps: Lessons learned moving machine-learning algorithms to production environments
There is a big difference between running a machine-learning algorithm manually from time to time and building a production system that runs thousands of machine-learning algorithms each day on petabytes of data, while also dealing with all the edge cases that arise. Robert Grossman discusses some of the lessons learned when building such a system and explores the tools that made the job easier.
| Talk Title | How to make analytic operations look more like DevOps: Lessons learned moving machine-learning algorithms to production environments |
| Speakers | Robert Grossman (University of Chicago) |
| Conference | Strata + Hadoop World |
| Conf Tag | Big Data Expo |
| Location | San Jose, California |
| Date | March 29-31, 2016 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Robert Grossman discusses some lessons learned moving machine-learning algorithms usually run manually by data scientists to operational environments where they run automatically on the new data that arrives each day. Robert offers three case studies in order to extract several techniques that have consistently proved useful and discuss how best these techniques can be used in practice: the first case study deals with the development of a system to analyze genomic datasets; the second describes the development of a system for the daily processing of new hyperspectral images to look for patterns of interest; and the third involves the incremental improvement of an algorithm for change detection run on streaming data. Topics from these case studies include:
