When one data center is not enough: Building large-scale stream infrastructures across multiple data centers with Apache Kafka

You may have successfully made the transition from single machines and one-off solutions to large, distributed stream infrastructures in your data center. But what if one data center is not enough? Ewen Cheslack-Postava explores resilient multi-data-center architecture with Apache Kafka, sharing best practices for data replication and mirroring as well as disaster scenarios and failure handling.

To manage the ever-increasing volume and velocity of data within your company, you may have successfully made the transition from single machines and one-off solutions to large, distributed stream infrastructures in your data center powered by Apache Kafka. But what’s to be done if one data center is not enough? Ewen Cheslack-Postava explores resilient multi-data-center architecture with Apache Kafka, sharing best practices for data replication and mirroring as well as disaster scenarios and failure handling. Ewen covers four scenarios—replication and failover for disaster recovery, data produced in one location but consumed in another, aggregate cluster for data analysis, and bidirection relication—discussing the requirements for each, providing a proven architecture, and explaining the benefits and limitations of the solution.