Managing data chaos in the world of microservices

When we talk about microservices, we usually focus on the communication layer. In practice, data is the much harder and often overlooked problem. Splitting applications into independent units leads to increased complexity, such as structural and semantic changes, knowledge sharing, and data discovery. Join Alexey Kachayev to explore emerging technologies created to tackle these challenges.

Microservices is one of the hottest topics in recent years, and the industry is shifting toward splitting applications into smaller and smaller independent units. This is all happening for very good reasons; you can gain a lot both in terms of technologies and organizational scalability. Many infrastructure tools to support the movement have been developed, from schedulers, deploy automation, and services discovery systems to development tools, like distribute tracers, log aggregators, and analyzers, and we’ve invented and reinvented protocols to make microservices communication even more efficient. However, one problem is often overlooked: the data layer is being diluted due to active encapsulation, which is essential for microservices to grow and evolve. As we move toward more independently encapsulated services, we’re experiencing dramatically increased challenges managing data, including: These problems are common, but most of our effort and attention is directed at infrastructure, which is easier to find generic solutions for. On the other hand, making sense of the data is hardly a generalizable problem. There have been many attempts to tame the chaos associated with independent dataset management. Alexey Kachayev discusses high-level approaches to build a sharable abstraction layer separating “physical” details from logical concerns as well as specific technologies you can leverage. The growing complexity of your data layer may overshadow the benefits of microservices architecture you deployed, so the sooner you start working on the solution, the easier it will be to manage the chaos.