Introducing Iceberg: Tables designed for object stores
Owen O'Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout with properties specifically designed for cloud object stores, such as S3. It provides a common set of capabilities such as partition pruning, schema evolution, atomic additions, removal, or replacements of files regardless of whether the data is stored in Avro, ORC, or Parquet.
| Talk Title | Introducing Iceberg: Tables designed for object stores |
| Speakers | Owen O’Malley (Cloudera), Ryan Blue (Netflix) |
| Conference | Strata Data Conference |
| Conf Tag | Make Data Work |
| Location | New York, New York |
| Date | September 11-13, 2018 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic. Netflix uses tables backed by S3 that, like other object stores, don’t fit this directory-based model: listings are much slower, renames are not atomic, and results are eventually consistent. Even tables in HDFS are problematic at scale, and reliable query behavior requires readers to acquire locks and wait. Owen O’Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout addresses the challenges of current Hive tables, with properties specifically designed for cloud object stores, such as S3. Iceberg is an Apache-licensed open source project. It specifies the portable table format and standardizes many important features, including:
