Real-time Hadoop: What an ideal messaging system should bring to Hadoop
Application messaging isnt newsolutions include IBM MQ, RabbitMQ, and ActiveMQ. Apache Kafka is a high-performance, high-scalability alternative that integrates well with Hadoop. Can modern distributed messaging systems like Kafka be considered a legacy replacement or is it purely complementary? Ted Dunning outlines Kafka's architectural benefits and tradeoffs to find the answer.
| Talk Title | Real-time Hadoop: What an ideal messaging system should bring to Hadoop |
| Speakers | Ted Dunning (MapR, now part of HPE) |
| Conference | Strata + Hadoop World |
| Conf Tag | Big Data Expo |
| Location | San Jose, California |
| Date | March 29-31, 2016 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Application developers and architects today are interested in making their applications as real-time as possible. To make an application respond to events as they happen, developers need a reliable way to move data as it is generated across different systems, one event at a time. In other words, these applications need messaging. Messaging solutions have existed for a long time. However, when compared to legacy systems, newer solutions like Apache Kafka offer higher performance, more scalability, and better integration with the Hadoop ecosystem. Kafka and similar systems are based on drastically different assumptions than legacy systems and have vastly different architectures. But do these benefits outweigh any tradeoffs in functionality? Ted Dunning dives into the architectural details and tradeoffs of both legacy and new messaging solutions to find the ideal messaging system for Hadoop. Topics include:
