Static and dynamic data with Postgres, Kafka, and Bottled Water

Chad Bailey demonstrates how to put the combination of static and dynamic data to work in your own apps.

Data stores like Postgres excel at storing and querying structured data, but many modern web apps have needs beyond what SQL can easily and quickly provide. Developers often find themselves writing the same data to multiple stores for searching, caching, or data warehousing. Meanwhile, tools like Kafka are giving developers a different way to think about their data. Instead of queries of static data, Kafka provides a real-time stream of app events that can be consumed for analysis and storage. Kafka emerged out of work at LinkedIn and is currently in use by companies like Netflix, PayPal, Spotify, and Uber. But what if you could view your Postgres database as a series of events, Kafka-style? That’s the goal of Bottled Water, an open source tool that has emerged from Confluent. Bottled Water uses features added in Postgres 9.4 to capture change events in your database and push them to a Kafka cluster for real-time analysis and processing. Heroku’s Chad Bailey walks attendees through building an end-to-end demo app to show how this all comes together. Chad demonstrates setting up a Kafka cluster and simple Ruby app to consume the Kafka event stream. Then, he’ll create a simple Rails app with a Postgres database to demonstrate Bottled Water. The demo itself is deceptively simple, but the real power comes from how you put the combination of static and dynamic data to work in your own apps. This session is sponsored by Heroku.