Strategies for integrating people and machine learning in online systems

Clara Labs is fusing machine learning (ML) with distributed human labor for natural language tasks. The result is a virtuous cycle: ML predictions improve workers efficiency, and workers help improve prediction models. Jason Laska explores the challenges of building a real-time(ish) knowledge workforce, how to integrate automation, and key strategies Clara Labs learned that enable scale.

Clara Labs is an email-based scheduling service for busy people. Simply “cc” Clara on an email to a person you want to meet with, and Clara handles the back-and-forth game of email tag for you. To build a robust and accurate system that gracefully handles nuanced requests, Clara Labs combined machine learning with a distributed human labor force. This service, available 24/7, consistently responds within 30 minutes or less and enables a single person to do work for an unbounded number of customers. A hybrid person-machine system has clear benefits, such as increased accuracy and decreased cost (i.e., increased scalability) via partial automation. Further, human input to the system leads to new annotations for retraining algorithms. There are great advantages to vertically integrating the ML annotation process directly with the product (e.g., the fidelity of labeled data increases when the annotator understands what actions will be derived directly from their work). Despite these advantages, there are several distinct challenges to building such a system: annotators are noisy and may be biased by bad ML predictions (if displayed). There also tends to be an inverse relationship between speed of data entry and annotator accuracy, and the learning curve for using a unique expert system may be high. In fact, simply measuring accuracy in the system may be challenging depending on time and cost constraints. Jason Laska explores the challenges of building a real-time(ish) knowledge workforce, how to integrate automation, and key strategies Clara Labs learned that enable scale. Along the way, Jason discusses incentives and algorithms for increasing both the accuracy and speed of human operators and for measuring their performance, strategies for dealing with task ambiguity, and tricks for building an effective ramping system to onboard workers. Jason also covers the “automation spectrum” (i.e., the integration points where machine learning predictions can be used to dramatically enhance human performance).