How Captricity built a human-level handwriting recognition engine using data-driven AI

Captricity has deployed a machine learning pipeline that can read handwriting at human-level accuracy. Ramesh Sridharan discusses the big ideas the company learned building and deploying this system, using data to identify specific problems to solve using AI and to evaluate and validate the algorithm itself and the overall system once deployed.

The last few years have seen an explosion in products and startups seeking to harness the promise of AI. Building such products presents many challenges during model development, from curating training data and investigating model architectures to training models. Once models are prototyped and developed, productionizing them is equally challenging: excellent performance on held-out test data does not always translate to production environments. Each of these steps can take weeks or even months and can cost tens of thousands of dollars. Captricity has iterated on this process through the development of many incremental models to solve the problem of recognizing structure and transcribing content of paper forms. This has culminated in a handwriting recognition system that achieves human-level accuracy and speed. A key observation for this approach is that not all problems require machine learning to solve on day one. Human feedback loops and reciprocal data applications are powerful tools to fill in gaps that ML can’t solve yet, and the judicious use of data can identify areas that maximize ML’s impact on a product while minimizing development time and effort. Similarly, human feedback loops are critical after deployment: a well-instrumented model with observable performance metrics is much easier to debug than an opaque one. For example, by identifying key metrics such as customer-level accuracy before deploying, Captricity was able to reduce the development lifecycle time from months to weeks and quickly identify failure cases to drive the next round of research improvements. Ramesh Sridharan outlines the journey from simple binary classification to handwriting recognition with near-human levels of accuracy at superhuman speeds and shares the successes and failures encountered on the road to this development. Along the way, he discusses the big ideas Captricity learned building and deploying this system, using data to identify specific problems to solve using AI and to evaluate and validate the algorithm itself and the overall system once deployed. Ramesh focuses on how Captricity used data to break down this large problem into manageable subproblems to solve with ML, how instrumenting an ML system with well-chosen metrics can enable debugging and implementation, reducing time to ship, and how the same infrastructure can enable ongoing evaluation and retraining.