The index as a first-class citizen

What happens when you take the index out of the database and make it a separate applicationperhaps one that is distributed, scalable, and takes full advantage of modern, multicore, high-memory hardware? Matthew Jaffee has spent the past few years finding out. He shares fruits of his labor: Pilosa, an open source distributed, sparse bitmap index.

A database is traditionally responsible for both storing and indexing data so that it can be kept safely and accessed quickly. Modern at-scale software architecture has increasingly tended toward breaking things apart (e.g., microservices), which databases have, with few exceptions, resisted. Matthew Jaffee offers an overview of Pilosa, an open source distributed, sparse bitmap index that exists as an acceleration layer over existing data stores, which is being successfully used in production to accelerate queries that were otherwise impractical due to high latency and excess memory use. Pilosa can be used to speed up certain queries to existing databases, or make joining data from multiple stores much faster. Matthew covers some background on databases and indexes and discusses the pros and cons of separating the index from the storage before diving into a general overview of Pilosa and a demonstration of how it can be used to reduce latency and enhance data exploration. This session is sponsored by Pilosa.