Transactional streaming: If you can compute it, you can probably stream it

In the race to pair streaming systems with stateful systems, the winners will be stateful systems that process streams natively. These systems remove the burden on application developers to be distributed systems experts and enable new applications to be both powerful and robust. John Hugg describes whats possible when integrated systems apply a transactional approach to event processing.

The operational systems of the future won’t look like those of the past, with authoritative databases, caching layers, and ORMs thrown in for fun. The distinction between traditional operational systems and event/stream processing has begun to blur. This event-stream focus loosens up processing constraints and makes scale easier to manage and reason about. Nevertheless, these systems still need state, and they still need to take action. The tools to build these kinds of operational systems are evolving but immature. Systems that focus on just streaming neglect state and vice versa. Cobbled-together hybrid systems offer flexibility but are complex to develop and deploy, may mask bugs, and display surprising behavior when components fail. The superficially “simple” task of connecting systems together requires both distributed systems expertise and tremendous effort to hammer down unforeseen production issues. John Hugg explores what’s possible when systems integrate event processing with state management in a consistent, transactional way where one event equals one ACID transaction. John covers a range of questions and topics. First, how does integration and atomic processing simplify failure management? How does this simplify building applications? How can users leverage stronger consistency to do more complex math and calculations within event processing problems? How can we move from struggling to count to dynamic counting and aggregation in real time? Many streaming systems focus on at-least-once or at-most-once delivery. Those that offer stronger guarantees are often very limited in how they interact with state. Can stronger consistency help achieve exactly-once semantics? John also discusses why the latency reduction from integration can mean the difference between decisions that control how the event is processed and reactive decisions that only affect the future and explains how, complementary to latency, integration can also increase throughput, which can mean the difference between managing a handful of servers or a fleet. This session is sponsored by VoltDB.