Backing off toward simplicity: Understanding the limits of deep learning

Deep learning is used broadly at the forefront of research, achieving state-of-the-art results across a variety of domains. However, that doesn't mean it's a fit for all tasksespecially when the constraints of production are considered. Stephen Merity investigates what tasks deep learning excels at, what tasks trigger a failure mode, and where current research is looking to remedy the situation.

Deep learning is used broadly at the forefront of research, achieving state-of-the-art results across a variety of domains. However, that doesn’t mean it’s a fit for all tasks—especially when the constraints of production are considered. While in some cases, deep learning can be applied without thought, most domains require understanding the task and the trade-offs involved when crafting a specific solution, especially when the system is designed with production in mind. Exploring successes in both research and production, Stephen Merity investigates what tasks deep learning excels at, what tasks trigger a failure mode, and where current research is looking to remedy the situation. By pulling apart specific examples, such as Google’s Neural Machine Translation architecture or Salesforce Research’s quasi-recurrent neural network, Stephen analyzes the trade-offs made when stepping away from research toward production systems, noting when deep learning is likely the wrong tool of choice, especially when factoring in real-world restrictions, such as training a custom model for each customer or tackling vast datasets.