Youre doing it wrong: How Zoomdata rearchitected streaming

The value of real-time streaming analytics with historical data is immense. Big data application Zoomdata updates historical dashboards in real time without complex reaggregations, but streaming in the age of the IoT requires handling of data in volumes not seen in traditional feeds. Erin Recachinas explains how Zoomdata moved to a scalable microservice architecture for streaming sources.

The ability to make quick decisions based on real-time data is important, but most business decisions aren’t made solely on a “window” of the most recent information. Analysts, executives, and decision makers everywhere want historical trends so they can look at what’s happening now in the context of historical data. Big data application Zoomdata brought streaming and historical data together, addressing the pie chart problem. However, streaming in the age of the IoT requires handling of data in volumes not seen in traditional feeds, and Zoomdata’s legacy streaming architecture wasn’t extensible enough to meet the emerging and future needs of this evolving space. Erin Recachinas explains how Zoomdata moved to a scalable microservice architecture for streaming sources, covering the value of streaming with historical data, what Zoomdata didn’t think about the first time around, and the new architecture for streaming analytics, from the user experience to the details of the implementation. This modern streaming architecture builds a table on the fly based on the shape of the data, writes as a stream to that table, then pushes the new data to visualizations without complex reaggregations, enabling users to view real-time metrics side by side with their historical data quickly and efficiently.