Designing a reactive data platform: Challenges, patterns, and antipatterns

Extracting value from data goes beyond robust ingestion pipelines and flexible storage. A solid data architecture addresses the needs of analysts and engineers by providing a simple, self-service ecosystem capable of handling any workload. Alex Silva discusses how the platform team at Pluralsight has been conquering these challenges by designing a platform using distributed, reactive services.

The last few years have seen a tremendous surge in data volume, along with an unparalleled explosion of toolsets and solutions aimed at extracting the most value from this deluge. Integrating these different technologies in a way that makes sense to the organization is a real challenge that has trampled many experienced engineering teams. Alex Silva discusses these challenges—their definition, mitigation, and potential solutions—and explains what makes a good design pattern (and what doesn’t) when architecting an integrated data platform. Alex covers the key architectural decisions Pluralsight made as it moved from a blank slate to a fully reactive self-service platform that is able to fulfill several business use cases, ranging from data ingestion to analysis, at both real-time and batch scales. Pluralsight’s current implementation consists of discrete microservices running on top of the JVM, using a mixture of Scala, Akka, and Java. Alex shares Pluralsight’s event-driven, reactive design that leverages REST, Hypermedia, and several open source frameworks and platforms, including Spring, Hadoop, YARN, Kafka, Storm, and Spark. Topics include: