Apache Spark is an amazing distributed system, but part of the bargain we've made with the infrastructure deamons involves providing the correct set of magic numbers (aka tuning) or our jobs may be eaten by Cthulhu. Holden Karau, Rachel Warren, and Anya Bida explore auto-tuning jobs using systems like Apache BEAM, Mahout, and internal Spark ML jobs as workloads.

Cloud native means designing software explicitly for the cloud, not trying to deploy to the cloud in retrospect - shoving a single replica of a monolith into Kubernetes wont cut it. Developing for …

Building a container network that is reliable, fast and easy to operate has become increasingly important in DigitalOceans distributed systems running on Kubernetes. Todays container networking tech …

Join members of the NATS team to learn about NATS Messaging!Distributed systems architecture has been disrupted via decomposition as cloud technology has matured - a move from monolithic applications …

Tim Berglund leads a basic architectural introduction to Kafka and walks you through using Kafka Streams and KSQL to process streaming data.

LinkedIn runs more than 1,800+ Kafka brokers that deliver more than two trillion messages a day. Running Kafka at such a scale makes automated operations a necessity. Jiangjie Qin shares lessons learned from operating Kafka at scale with minimum human intervention.

In distributed systems, some business transactions and even more end-to-end processes stretch across boundaries of individual services. While event-driven choreography leads to nicely decoupled systems, complex event chains cause headaches. Bernd Rcker explains why transforming certain events into commands is beneficial and how to avoid losing sight of larger-scale flows.

Mario-Leander Reimer explores key JEE technologies that can be used to build JEE-powered data services and walks you through implementing the individual data processing tasks of a simplified showcase application. You'll then deploy and orchestrate the individual data services using OpenShift, illustrating the scalability of the overall processing pipeline.

Kubernetes Quality of Service (QoS) offers powerful primitives for resource management, that is workload prioritization, fairness and efficiency. But it's also a complex topic to understand and get ri …

In the complex world of microservices and distributed systems, we need to understand what our software is doing. Traditional tools, such as logs, read by humans and filtered by crude rules, arent powerful enough. Sam Stokes explains that we need new, better tools and why this will also require us to design our systems to give the tools better data.

Cornelia Davis explains how to use an event-driven approach to address the fallacies of distributed computing in a very different way, offering significant benefits over request-response, and details event-oriented solutions to problems commonly addressed with well-known patterns.