Toppling the mainframe: Enterprise-grade streaming under 2 ms on Hadoop

What if we have reached the point where open source can handle massively difficult streaming problems with enterprise-grade durability? Ilya Ganelin presents Capital Ones novel solution for real-time decisioning on Apache Apex. Ilya shows how Apex provides unique capabilities that ensure less than 2 ms latency in an enterprise-grade solution on Hadoop.

These days everyone is excited about big data and fast data. Capital One has embraced this new generation of technology with open arms. However, as Edward Heinlein was fond of reminding us, “TANSTAFL—There ain’t no such thing as a free lunch.” For many years, there’s been a very real battle around the standard operating model of software. Tech giants like Oracle and IBM have traditionally built massively expensive enterprise-ready products, while the open source community provides free, albeit usually inferior, software. For a product to be enterprise-ready, it must guarantee complete reliability alongside performance and flexibility. There are notable successes in the open source world such as Linux and Open SSH/SSL, but the realm of distributed stream computing has lacked comparable solutions. Capital One set out to find whether we could build or find enterprise-ready technology in the open source world to tackle difficult streaming problems that also provides equivalent performance, durability, and availability as a mainframe computer. Ilya Ganelin details Capital One’s attempt to answer this question in a rigorous and complete way, not just by making a prototype or discovering exciting new tools, but by creating an open source-based, enterprise-ready product that can transparently replace an enormously expensive proprietary solution. Ilya presents Capital One’s novel solution for real-time decisioning on Apache Apex. Topics include: