Stream all the things!

"Stream" is a buzzword for several things that share the idea of timely handling of neverending data. Big data architectures are evolving to be stream oriented. Microservice architectures are inherently message driven. Dean Wampler defines "stream" based on characteristics for such systems, using specific tools as examples, and argues that big data and microservices architectures are converging.

Big data started with an emphasis on batch-oriented architectures, where data is captured in large, scalable stores then processed using batch jobs. To reduce the gap between data arrival and information extraction, these architectures are now evolving to be stream oriented, where data is processed as it arrives. (Fast data is the new buzzword for this process.) Microservices are inherently message driven, a core tenet of reactive systems, responding to requests for service and sending messages to other microservices in turn. Hence, they are also stream oriented, in a sense. Because it’s trendy, the word “stream” is used in both spheres, because both are concerned with a neverending sequence of data, but the resemblance is not superficial. Many of the same challenges and design patterns are shared. Hence, the movement to stream-oriented architectures is driving a convergence of data-centric and microservice architectures. Dean Wampler defines “stream” based on characteristics for such systems, using specific tools as examples, and argues that big data and microservices architectures are converging. Dean begins by quantifying what streaming means in the context of four axes of concern that cross the fast data and microservice divide: Dean then considers specific examples of streaming tools and explains how they fit on these axes, including heavy hitters in the data world, such as Spark and Kafka, as well as microservice toolkits, such as Akka and Rx. Dean concludes by speculating on the future of these trends—sharing his belief that fast data and microservice architectures will converge, driven by the ever-growing importance of data and the scalability of fast data streaming.