hadoop

Open Source at Uber

Open Source at Uber

February 7, 2020

Uber officially launched its open source journey in 2015 which started with three passionate engineers. In only three years, Ubers open source program has grown and fostered several outstanding open …
Open Source at Uber

Open Source at Uber

February 3, 2020

Uber officially began its open source journey in 2015 when three passionate engineers decided to contribute Ubers work back to the community. In only three years, Ubers open source program has foste …
Clouds and containers: Case studies for big data

Clouds and containers: Case studies for big data

January 25, 2020

Once the data has been captured, how can the cloud, containers, and a data fabric combine to build the infrastructure to provide the business insights? Paul Curtis explores three customer deployments that leverage the best of the private clouds and containers to provide a flexible big data environment.
Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)

Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)

January 25, 2020

The days of deploying Java code to Hadoop and Spark data lakes for data science and ML are numbered. Welcome to the future. Containers and Kubernetes make great language-agnostic distributed computing clusters: it's just as easy to deploy Python as it is Java. Mathew Lodge shows you how.
Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)

Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)

January 25, 2020

The days of deploying Java code to Hadoop and Spark data lakes for data science and ML are numbered. Welcome to the future. Containers and Kubernetes make great language-agnostic distributed computing clusters: it's just as easy to deploy Python as it is Java. Mathew Lodge shows you how.
Data discovery and lineage: Integrating streaming data in the public cloud with on-prem, classic data stores, and heterogeneous schema types

Data discovery and lineage: Integrating streaming data in the public cloud with on-prem, classic data stores, and heterogeneous schema types

January 25, 2020

Comcasts streaming data platform comprises ingest, transformation, and storage services in the public cloud, with Apache Atlas for data discovery and lineage. Barbara Eckman explains how Comcast recently integrated on-prem data sources, including traditional data warehouses and RDBMSs, which required its data governance strategy to include relational and JSON schemas in addition to Apache Avro.
Deep learning on YARN: Running distributed TensorFlow, MXNet, Caffe, and XGBoost on Hadoop clusters

Deep learning on YARN: Running distributed TensorFlow, MXNet, Caffe, and XGBoost on Hadoop clusters

January 24, 2020

In order to train deep learning and machine learning models, you must leverage applications such as TensorFlow, MXNet, Caffe, and XGBoost. Wangda Tan discusses new features in Apache Hadoop 3.x to better support deep learning workloads and demonstrates how to run these applications on YARN.
Feet on the ground, head in the clouds (sponsored by AtScale)

Feet on the ground, head in the clouds (sponsored by AtScale)

January 23, 2020

Interested in how Ebates is using a hybrid on-premises and cloud implementation to scale out its centralized business intelligence and data hub? Mark Stange-Tregear shares the history, business context, and technical plan around Ebatess hybrid Hadoop-AWS cloud approach.
From flat files to deconstructed database: The evolution and future of the big data ecosystem

From flat files to deconstructed database: The evolution and future of the big data ecosystem

January 22, 2020

Big data infrastructure has evolved from flat files in a distributed filesystem to an efficient ecosystem to a fully deconstructed and open source database with reusable components. Julien Le Dem discusses the key open source components of the big data ecosystem and explains how they relate to each other and how they make the ecosystem more of a database and less of a filesystem.
Getting ready for GDPR: Securing and governing hybrid, cloud, and on-premises big data deployments, step by step

Getting ready for GDPR: Securing and governing hybrid, cloud, and on-premises big data deployments, step by step

January 22, 2020

New regulations such as GDPR are driving new compliance, governance, and security challenges for big data. Infosec and security groups must ensure a consistently secured and governed environment across multiple workloads. Mark Donsky, Syed Rafice, Mubashir Kazia, Ifigeneia Derekli, and Camila Hiskey share hands-on best practices for meeting these challenges, with special attention paid to GDPR.
Marmaray: A generic, scalable, and pluggable Hadoop data ingestion and dispersal framework

Marmaray: A generic, scalable, and pluggable Hadoop data ingestion and dispersal framework

January 20, 2020

Danny Chen, Omkar Joshi, and Eric Sayle offer an overview of Marmaray, a generic Hadoop ingestion and dispersal framework recently released to production at Uber. You'll learn how Marmaray can meet a team's data needs by ensuring that data can be reliably ingested into Hive or dispersed into online data stores and take a deep dive into the architecture to see how it all works.
Real-time analytics and BI with data lakes and data warehouses using Kudu, HBase, Spark, and Kafka: Lessons learned

Real-time analytics and BI with data lakes and data warehouses using Kudu, HBase, Spark, and Kafka: Lessons learned

January 19, 2020

Mauricio Aristizabal shares lessons learned from migrating Impact's traditional ETL platform to a real-time platform on Hadoop (leveraging the full Cloudera EDH stack). Mauricio also discusses the company's data lake in HBase, Spark Streaming jobs (with Spark SQL), using Kudu for "fast data" BI queries, and using Kafka's data bus for loose coupling between components.
TuneIn: How to get your jobs tuned while you are sleeping

TuneIn: How to get your jobs tuned while you are sleeping

January 17, 2020

Have you ever tuned a Spark or MR job? If the answer is yes, you already know how difficult it is to tune more than hundred parameters to optimize the resources used. Manoj Kumar, Pralabh Kumar, and Arpan Agrawal offer an overview of TuneIn, an auto-tuning tool developed to minimize the resource usage of jobs. Experiments have shown up to a 50% reduction in resource usage.
What's the Hadoop-la about Kubernetes?

What's the Hadoop-la about Kubernetes?

January 16, 2020

Kubernetes (K8s)the open source container orchestration system for modern big data workloadsis increasingly popular. While the promised land is a unified platform for cloud-native stateless and stateful data services, stateful, multiservice big data cluster orchestration brings unique challenges. Anant Chintamaneni and Nanda Vijaydev outline the considerations for big data services for K8s.
Building an Enterprise/Cloud Analytics Platform with Jupyter Enterprise Gateway

Building an Enterprise/Cloud Analytics Platform with Jupyter Enterprise Gateway

January 9, 2020

Data science and analytics departments are now common place for enterprises determined to maximize their operations. While Jupyter Notebooks have significantly decreased the cost of admission into this space, enterprises are finding that data science at scale is difficult within the current framework. Jupyter Enterprise Gateway is designed to address these scalability issues for the enterprise.
Pangeo: Big data climate science in the cloud

Pangeo: Big data climate science in the cloud

January 6, 2020

Climate science is being flooded with petabytes of data, overwhelming traditional modes of data analysis. The Pangeo project is building a platform to take big data climate science into the cloud using SciPy and large-scale interactive computing tools. Join Ryan Abernathey and Yuvi Panda to find out what the Pangeo team is building and why and learn how to use it.
Distributed TensorFlow on Hops

Distributed TensorFlow on Hops

December 30, 2019

Fabio Buso offers demonstrations of frameworks for building distributed TensorFlow applications on the Hops platform and walks you through the whole model lifecycle, from debugging and visualizing models on TensorBoard to parallel experimentation and distributed training (with the help of Spark) to model deployment and inferencing using TensorFlow Serving and Kubernetes.
Improving performance with Tesser

Improving performance with Tesser

December 22, 2019

Kyle Kingsbury offers an overview of Tesser, a Clojure library for writing commutative, parallel folds that can be chained and composed into complex single-pass reductions that are dramatically faster on multicore systems and can be transparently distributed over Hadoop.
Architecting data platforms for cybersecurity

Architecting data platforms for cybersecurity

December 12, 2019

Data is becoming a crucial weapon to secure an organization against cyber threats. Charaka Goonatilake shares strategies for designing effective data platforms for cybersecurity using big data technologies, such as Spark and Hadoop, and explains how these platforms are being used in real-world examples of data-driven security.
Audi's journey to an enterprise big data platform

Audi's journey to an enterprise big data platform

December 12, 2019

Carsten Herbe and Matthias Graunitz detail Audi's journey from a Hadoop proof of concept to a multitenant enterprise platform, sharing lessons learned, the decisions Audi made, and how a number of use cases are implemented using the platform.
Autonomous ETL with materialized views

Autonomous ETL with materialized views

December 11, 2019

Adesh Rao and Abhishek Somani share a framework for materialized views in SQL-on-Hadoop engines that automatically suggests, creates, uses, invalidates, and refreshes views created on top of data for optimal performance and strict correctness.
Batch and real-time processing in LINE's log analysis platform

Batch and real-time processing in LINE's log analysis platform

December 11, 2019

LINEone of the most popular messaging applications in Asiaoffers many services, such as its news application. These services sometimes depend on real-time processing. Wataru Yukawa offers an overview of LINE's web tracking system, which consists of the JavaScript SDK, NGINX Fluentd, Kafka, Elasticsearch, and Hadoop, and explains how it helps with batch and real-time processing.
Executive Briefing: BI on big data

Executive Briefing: BI on big data

December 10, 2019

If your goal is to provide data to an analyst rather than a data scientist, whats the best way to deliver analytics? There are 70+ BI tools in the market and a dozen or more SQL- or OLAP-on-Hadoop open source projects. Mark Madsen and Shant Hovsepian discuss the trade-offs between a number of architectures that provide self-service access to data.
Fast analytics on fast data: Kudu as a storage layer for banking applications

Fast analytics on fast data: Kudu as a storage layer for banking applications

December 9, 2019

Olaf Hein explains how a large German bank relies on a Kudu-based data platform to speed up business processes. Olaf highlights key data access patterns and the system architecture and shares best practices and lessons learned using Kudu in development and operations.
Hadoop under attack: Securing data in a banking domain

Hadoop under attack: Securing data in a banking domain

December 9, 2019

The apparent difficulty of managing Hadoop compared to more traditional and proprietary data products makes some companies wary of the Hadoop ecosystem, but managing security is becoming more accessible in the Hadoop space, particularly in the Cloudera stack. Federico Leven offers an overview of an end-to-end security deployment on Hadoop and the data and security governance policies implemented.
How to protect big data in a containerized environment

How to protect big data in a containerized environment

December 9, 2019

Recent headline-grabbing data breaches demonstrate that protecting data is essential for every enterprise. The best-of-breed approach for big data is HDFS configured with Transparent Data Encryption (TDE), but TDE can be difficult to configure and manageissues that are only compounded when running on Docker containers. Thomas Phelan discusses these challenges and explains how to overcome them.
Scaling data science (teams and technologies)

Scaling data science (teams and technologies)

December 6, 2019

Cox Automotive is the worlds largest automotive service organization, which means it can combine data from across the entire vehicle lifecycle. Cox is on a journey to turn this data into insights. David Asboth and Shaun McGirr share their experience building up a data science team at Cox and scaling the company's data science process from laptop to Hadoop cluster.
Scaling the AI hierarchy of needs with TensorFlow, Spark, and Hops

Scaling the AI hierarchy of needs with TensorFlow, Spark, and Hops

December 5, 2019

Distributed deep learning can increase the productivity of AI practitioners and reduce time to market for training models. Hadoop can fulfill a crucial role as a unified feature store and resource management platform for distributed deep learning. Jim Dowling offers an introduction to writing distributed DL applications, covering TensorFlow and Apache Spark frameworks that make distribution easy.