Rethinking microservices with stateful streams

Ben Stopford looks at two forces that sit in opposition: data systems (which focus on exposing data) and services (which focus on encapsulating it). How should we balance these two? Streaming offers a solution.

When building service-based systems, we don’t generally think too much about data. If we need data from another service, we ask for it. This pattern works well for whole swathes of use cases, particularly ones where datasets are small and requirements are simple. But real business services have to join and operate on datasets from many different sources. This can be slow and cumbersome in practice. These problems stem from an underlying dichotomy. Data systems are built to make data as accessible as possible—a mindset that focuses on getting the job done. Services, instead, focus on encapsulation—a mindset that allows independence and autonomy as we evolve and grow. But these two forces inevitably compete in most serious service-based architectures. Ben Stopford explains why understanding and accepting this dichotomy is an important part of designing service-based systems at any significant scale. Ben looks at how companies use log-backed architectures to build an immutable narrative that balances data that sits inside their services with data that is shared, an approach that allows the likes of Uber, Netflix, and LinkedIn to scale to millions of events per second. Ben concludes with a set implementation patterns, starting lightweight and gradually getting more functional, paving the way for an evolutionary approach to building log-backed microservices.