hadoop

Analytics at ING: Technology solutions to create a real-time, data-driven bank

Analytics at ING: Technology solutions to create a real-time, data-driven bank

January 2, 2020

Bas Geerdink explains why and how ING is becoming more and more data-driven, sharing use cases, architecture, and technology choices along the way.
Big data on the rise: Views of emerging trends and predictions from real-life end users

Big data on the rise: Views of emerging trends and predictions from real-life end users

January 2, 2020

John Mertic and Cupid Chan share real end-user perspectives from companies like GE on how they are using big data tools, challenges they face, and where they are looking to focus investmentsall from a vendor-neutral viewpoint.
Big telco real-time network analytics

Big telco real-time network analytics

January 2, 2020

Data transfer is one of the most pressing problems for telecom companies, as cost increases in tandem with the growing data requirements. Yousun Jeong details how SKT has dealt with this problem.
LINE's log analysis platform

LINE's log analysis platform

December 30, 2019

Data is a very important asset to LINE, one of the most popular messaging applications in Asia. Wataru Yukawa explains how LINE gets the most out of its data using a Hadoop data lake and an in-house log analysis platform.
Real-world patterns for continuously deployed advanced analytics

Real-world patterns for continuously deployed advanced analytics

December 30, 2019

How can we drive more data pipelines, advanced analytics, and machine learning models into production? How can we do this both faster and more reliably? Graham Gear draws on real-world processes and systems to explain how it's possible to apply continuous delivery techniques to advanced analytics, realizing business value earlier and more safely.
Smart cities, the smart grid, the IoT, and big data

Smart cities, the smart grid, the IoT, and big data

December 29, 2019

Smart cities and the electricity smart grid have become leading examples of the IoT, in which distributed sensors describe mission-critical behavior by generating billions of metrics daily. Mark Donsky and Syed Rafice show how smart utilities and cities rely on Hadoop to capture, analyze, and harness this data to increase safety, availability, and efficiency across the entire electricity grid.
An architecture for merging fast data and enterprise applications: The SMACK stack

An architecture for merging fast data and enterprise applications: The SMACK stack

December 28, 2019

Big data architectures and enterprise/microservice architectures are slowly converging. Big data is transitioning to "fast data," emphasizing streaming over batch processing, while data processing is growing ubiquitous. Dean Wampler explores the SMACK stackSpark, Mesos, Akka, Cassandra, and Kafkaand explains how it addresses the needs of both fast data and the enterprise.
Architects as change agents: The case of Apache HBase adoption at Visa

Architects as change agents: The case of Apache HBase adoption at Visa

December 28, 2019

A redesign of the entire data platform behind Visa's business is underway. The Apache Hadoop ecosystem is becoming a staple of many of its solutions, requiring architects to evolve existing live applications as well as the processes and culture of engineering at Visa. Partha Saha explores Apache HBase adoption at Visa and discusses how architects are leading the way.
The trials of machine learning at Zendesk

The trials of machine learning at Zendesk

December 28, 2019

Simply building a successful machine learning product is extremely challenging, and just as much effort is needed to turn that model into a customer-facing product. Drawing on their experience working on Zendesk's article recommendation product, Wai Yau and Jeffrey Theobald discuss design challenges and real-world problems you may encounter when building a machine learning product at scale.
Train, predict, and serve: How to put your machine learning model into production

Train, predict, and serve: How to put your machine learning model into production

December 28, 2019

Aki Ariga explains how to put your machine learning model into production, discusses common issues and obstacles you may encounter, and shares best practices and typical architecture patterns of deployment ML models with example designs from the Hadoop and Spark ecosystem using Cloudera Data Science Workbench.
A practitioners guide to securing your Hadoop cluster

A practitioners guide to securing your Hadoop cluster

December 16, 2019

Many Hadoop clusters lack even basic security controls. Michael Yoder, Ben Spivey, Mark Donsky, and Mubashir Kazia walk you through securing a Hadoop cluster. You'll start with a cluster with no security and then add security features related to authentication, authorization, encryption of data at rest, encryption of data in transit, and complete data governance.
Architecting for change: LinkedIn's new data ecosystem

Architecting for change: LinkedIn's new data ecosystem

December 16, 2019

Shirshanka Das and Yael Garten describe how LinkedIn redesigned its data analytics ecosystem in the face of a significant product rewrite, covering the infrastructure changes, such as client-side activity tracking, a unified reporting platform, and data virtualization techniques to simplify migration, that enable LinkedIn to roll out future product innovations with minimal downstream impact.
Authorization in the cloud: Enforcing access control across compute engines

Authorization in the cloud: Enforcing access control across compute engines

December 16, 2019

Li Li and Hao Hao elaborate the architecture of Apache Sentry + RecordService for Hadoop in the cloud, which provides unified, fine-grained authorization via role- and attribute-based access control, to encourage attendees to adopt Apache Sentry and RecordService to protect sensitive data on the multitenant cloud across the Hadoop ecosystem.
Beyond Hadoop at Yahoo: Interactive analytics with Druid

Beyond Hadoop at Yahoo: Interactive analytics with Druid

December 16, 2019

Himanshu Gupta explains why Yahoo has been increasingly investing in interactive analytics and how it leverages Druid to power a variety of internal- and external-facing data applications.
Big data architectural patterns and best practices on AWS

Big data architectural patterns and best practices on AWS

December 15, 2019

Siva Raghupathy demonstrates how to use Hadoop innovations in conjunction with Amazon Web Services (cloud) innovations.
Big data processing with Hadoop and Spark, the Uber way

Big data processing with Hadoop and Spark, the Uber way

December 15, 2019

Praveen Murugesan explains how Uber leverages Hadoop and Spark as the cornerstones of its data infrastructure. Praveen details the current data architecture at Uber and outlines some of the unique challenges with data processing Uber faced as well as its approach to solving some key issues in order to continue to power Uber's real-time marketplace.
Breeding data scientists: A four-year study

Breeding data scientists: A four-year study

December 15, 2019

At Strata + Hadoop World 2012, Amy O'Connor and her daughter Danielle Dean shared how they learned and built data science skills at Nokia. This year, Amy and Danielle explore how the landscape in the world of data science has changed in the past four years and explain how to be successful deriving value from data today.
Driving open source adoption within the enterprise

Driving open source adoption within the enterprise

December 13, 2019

Theres been much discussion on open source versus commercial; CIOs and CTOs are increasingly interested in solutions that blend the benefits of both worlds. Ron Bodkin explains how Teradata drives open source adoption inside enterprises through a range of initiatives: direct contributions to open source projects, building orchestration software, and providing technical expertise.
Elastic data services on Mesos via Mesospheres DC/OS

Elastic data services on Mesos via Mesospheres DC/OS

December 13, 2019

Adam Bordelon and Mohit Soni demonstrate how projects like Apache Myriad (incubating) can install Hadoop on Mesosphere DC/OS alongside other data center-scale applications, enabling efficient resource sharing and isolation across a variety of distributed applications while sharing the same cluster resources and hence breaking silos.
Hadoop and Spark at ING: An overview of the architecture, security, and business cases at a large international bank

Hadoop and Spark at ING: An overview of the architecture, security, and business cases at a large international bank

December 12, 2019

Bas Geerdink offers an overview of the evolution that the Hadoop ecosystem has taken at ING. Since 2013, ING has invested heavily in a central data lake and data management practice. Bas shares historical lessons and best practices for enterprises that are incorporating Hadoop into their infrastructure landscape.
Hadoop application architectures: Architecting a next-generation data platform for real-time ETL, data analytics, and data warehousing

Hadoop application architectures: Architecting a next-generation data platform for real-time ETL, data analytics, and data warehousing

December 12, 2019

Jonathan Seidman, Gwen Shapira, Mark Grover, and Ted Malaska demonstrate how to architect a modern, real-time big data platform and explain how to leverage components like Kafka, Impala, Kudu, Spark Streaming, and Spark SQL with Hadoop to enable new forms of data processing and analytics such as real-time ETL, change data capture, and machine learning.
Hadoop in the cloud: A Nielsen use case

Hadoop in the cloud: A Nielsen use case

December 12, 2019

Cloudera CEO Tom Reilly and James Powell, global CTO of Nielsen, discuss the dynamics of Hadoop in the cloud, what to consider at the start of the journey, and how to implement a solution that delivers flexibility while meeting key enterprise requirements.
How a Spark-based feature store can accelerate big data adoption in financial services

How a Spark-based feature store can accelerate big data adoption in financial services

December 12, 2019

Kaushik Deka and Phil Jarymiszyn discuss the benefits of a Spark-based feature store, a library of reusable features that allows data scientists to solve business problems across the enterprise. Kaushik and Phil outline three challenges they facedsemantic data integration within a data lake, high-performance feature engineering, and metadata governanceand explain how they overcame them.
How the largest US healthcare dataset in Hadoop enables patient-level analytics in near real time

How the largest US healthcare dataset in Hadoop enables patient-level analytics in near real time

December 12, 2019

The need to find efficiencies in healthcare is becoming paramount as our society and the global population continue to grow and live longer. Navdeep Alam shares his experience and reviews current and emerging technologies in the marketplace that handle working with unbounded, de-identified patient datasets in the billions of rows in an efficient and scalable way.
Lessons learned running Hadoop and Spark in Docker

Lessons learned running Hadoop and Spark in Docker

December 11, 2019

Many initiatives for running applications inside containers have been scoped to run on a single host. Using Docker containers for large-scale environments poses new challenges, especially for big data applications like Hadoop. Thomas Phelan shares lessons learned and some tips and tricks on how to Dockerize your big data applications in a reliable, scalable, and high-performance environment.
Model visualization

Model visualization

December 11, 2019

Though visualization is used in data science to understand the shape of the data, it's not widely used for statistical models, which are evaluated based on numerical summaries. Amit Kapoor explores model visualization, which aids in understanding the shape of the model, the impact of parameters and input data on the model, the fit of the model, and where it can be improved.
Powering real-time analytics on Xfinity using Kudu

Powering real-time analytics on Xfinity using Kudu

December 10, 2019

Sridhar Alla and Kiran Muglurmath explain how real-time analytics on Comcast Xfinity set-top boxes (STBs) help drive several customer-facing and internal data-science-oriented applications and how Comcast uses Kudu to fill the gaps in batch and real-time storage and computation needs, allowing Comcast to process the high-speed data without the elaborate solutions needed till now.
SAS: More open than you might think

SAS: More open than you might think

December 10, 2019

Hadoop and its ecosystem have changed analytics profoundly. Paul Kent offers an overview of SAS's participation in open platforms and introduces SAS Viya, a new unified and open analytics architecture that lets you scale analytics in the cloud and code as you choose.