Kafka at PayPal: Enabling 400 billion messages a day

PayPal is one of the biggest Kafka users in the industry; it manages and maintains over 40 production Kafka clusters in three geodistributed data centers and supports 400 billion Kafka messages a day. Kevin Lu, Maulin Vasavada, and Na Yang explore the management and monitoring PayPal applies to Kafka, from client-perceived statistics to configuration management, failover, and data loss auditing.

Kafka is playing an increasingly important role in messaging and streaming systems. Managing fast-growing Kafka deployments and supporting customers with various requirements can become a challenging task for a small team of only a few engineers. The availability of the Kafka infrastructure is essential to PayPal’s revenue stream. The company needs to catch issues before systems break down, know exactly how available it is for each client, and preemptively recover from problems when they occur. It also needs to have a clear view of message loss in its end-to-end Kafka pipeline. Operational tooling is critical to PayPal’s success, and the company has developed tools such as data loss auditing, full and partial cluster failovers, client and server-side KPI measurements, and a control panel for Kafka clusters. Kevin Lu, Maulin Vasavada, and Na Yang explore the management and monitoring PayPal applies to Kafka, from client-perceived statistics to configuration management, failover, and data loss auditing. You’ll discover the criticality of a large-scale Kafka environment, the set of tools you’ll need to make this environment work and supportable, and how to provide the performance and scalability needed for PayPal’s data volume and SLA. Along the way, they highlight the architecture of PayPal’s next-generation Kafka monitoring and management system, built for serving all the Kafka-as-a-service needs.