An architecture for merging fast data and enterprise applications: The SMACK stack

Big data architectures and enterprise/microservice architectures are slowly converging. Big data is transitioning to "fast data," emphasizing streaming over batch processing, while data processing is growing ubiquitous. Dean Wampler explores the SMACK stackSpark, Mesos, Akka, Cassandra, and Kafkaand explains how it addresses the needs of both fast data and the enterprise.

Big data architectures—those using large frameworks like Spark, YARN, HBase or Cassandra, HDFS, and Kafka—have been slow to embrace microservices. Everything else—i.e., enterprise architectures (whether microservice-based or not)—have been less concerned with large data volumes and more interested in reactive tools that are flexible, adaptive, scalable, resilient, and event/message driven. These two spheres are now slowly converging, as data teams need answers faster (hence the growing interest in streaming architectures) and enterprises become more data driven (hence the need for sophisticated, scalable data processing options that are still event-driven). Another trend affecting both spheres is the need to optimize resource utilization and lower costs, which has led to the growth of virtualized services on flexible, efficient clusters that are capable of running all services. While Hadoop has dominated the data world, it is a first-generation architecture that isn’t well suited for more general enterprise needs. Dean Wampler explores the SMACK stack—Spark, Mesos, Akka, Cassandra, and Kafka—discussing the role each tool plays in addressing the needs of fast data and enterprise environments, as well as what’s missing and what areas need to mature.