How to achieve zero-latency IoT and FSI data processing with Spark
Yaron Haviv explains how to design real-time IoT and FSI applications, leveraging Spark with advanced data frame acceleration. Yaron then presents a detailed, practical use case, diving deep into the architectural paradigm shift that makes the powerful processing of millions of events both efficient and simple to program.
| Talk Title | How to achieve zero-latency IoT and FSI data processing with Spark |
| Speakers | |
| Conference | Strata + Hadoop World |
| Conf Tag | Make Data Work |
| Location | New York, New York |
| Date | September 27-29, 2016 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
IoT and financial trading platforms share some commonality: they intercept massive amounts of sensor or event data and must provide insights and actions in real time. The technology challenge is huge since we need to combine fast event streams, historical state, and consistent data update transactions (e.g., time series data and statistical aggregators) with data science and machine learning and present results through real-time dashboards or drive immediate corrective actions. Traditional solutions like the Lambda Architecture or stream processing can’t meet the requirement, but with the latest advancements in Spark 2.0 coupled with Data Frames coprocessing in external real-time engines, we can analyze millions of complex events per second and deliver true real-time dashboards or actionable insights. Yaron Haviv explains how to design real-time IoT and FSI applications, leveraging Spark with advanced data frame acceleration. Yaron then presents a detailed, practical use case, diving deep into the architectural paradigm shift that makes the powerful processing of millions of events both efficient and simple to program.
