HopsWorks: Multitenant Hadoop as a service

Currently, multitenancy in Hadoop is limited to organizations running separate Hadoop clusters, and the secure sharing of resources is achieved using virtualization or containers. Jim Dowling describes how HopsWorks enables organizations to securely share a single Hadoop cluster using projects and a new metadata layer that enables protection domains while still allowing data sharing.

Jim Dowling describes how HopsWorks enables organizations to securely share a single Hadoop cluster using projects and a new metadata layer that enables protection domains while still allowing data sharing. HopsWorks is a frontend to Hadoop that provides a new model for multitenancy in Hadoop, based around projects. A project is like a GitHub project: the owner of the project manages membership, and users can be one of two roles in the project—data scientists, who can just run programs, and data owners, who can also curate, import, and export data. Users can’t copy data between projects or run programs that process data from different projects, even if the user is a member of multiple projects. That is, we implement multitenancy with dynamic roles, where the user’s role is based on the currently active project. (Users can still share datasets between projects, however.) HopsWorks also supports Apache Zeppelin with access control, free-text search for files in HDFS using Elasticsearch, and a metadata designer tool that enables users to curate files and directories in HDFS. HopsWorks has been enabled by migrating all metadata in HDFS and YARN into an open source, shared nothing, in-memory, distributed database called NDB. HopsWorks is open source and licensed as Apache v2, with database connectors licensed as GPL v2. From late January 2016, HopsWorks will be provided as software as a service for researchers and companies in Sweden by the Swedish ICT SICS Data Center.