cluster

Choice Hotels's journey to better understand its customers through self-service analytics

Choice Hotels's journey to better understand its customers through self-service analytics

December 14, 2019

Narasimhan Sampath and Avinash Ramineni share how Choice Hotels International used Spark Streaming, Kafka, Spark, and Spark SQL to create an advanced analytics platform that enables business users to be self-reliant by accessing the data they need from a variety of sources to generate customer insights and property dashboards and enable data-driven decisions with minimal IT engagement.
Creating and evaluating a distance measure

Creating and evaluating a distance measure

December 14, 2019

Whether we're talking about spam emails, merging records, or investigating clusters, there are many times when having a measure of how alike things are makes them easier to work with (e.g., with unstructured data that isn't incorporated into your data models). Melissa Santos offers a practical approach to creating a distance metric and validating with business owners that it provides value.
Elastic data services on Mesos via Mesospheres DC/OS

Elastic data services on Mesos via Mesospheres DC/OS

December 13, 2019

Adam Bordelon and Mohit Soni demonstrate how projects like Apache Myriad (incubating) can install Hadoop on Mesosphere DC/OS alongside other data center-scale applications, enabling efficient resource sharing and isolation across a variety of distributed applications while sharing the same cluster resources and hence breaking silos.
Model visualization

Model visualization

December 11, 2019

Though visualization is used in data science to understand the shape of the data, it's not widely used for statistical models, which are evaluated based on numerical summaries. Amit Kapoor explores model visualization, which aids in understanding the shape of the model, the impact of parameters and input data on the model, the fit of the model, and where it can be improved.
Securing Apache Kafka

Securing Apache Kafka

December 10, 2019

With Apache Kakfa 0.9, the community has introduced a number of features to make data streams secure. Jun Rao explains the motivation for making these changes, discusses the design of Kafka security, and demonstrates how to secure a Kafka cluster. Jun also covers common pitfalls in securing Kafka and talks about ongoing security work.
Stream analytics in the enterprise: A look at Intels internal IoT implementation

Stream analytics in the enterprise: A look at Intels internal IoT implementation

December 10, 2019

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was designed as a multitenant platform with built-in analytical capabilities and based on open source big data technologies and containers. Moty highlights the lessons learned from this journey with a thorough review of the platforms architecture.
Tuning Spark machine-learning workloads

Tuning Spark machine-learning workloads

December 8, 2019

Spark's efficiency and speed can help reduce the TCO of existing clusters. This is because Spark's performance advantages allow it to complete processing in drastically shorter batch windows with higher performance per dollar. Raj Krishnamurthy offers a detailed walk-through of an alternating least squares-based matrix factorization workload able to improve runtimes by a factor of 2.22.
Turning petabytes of data into millions in cost savings for the worlds biggest retailers

Turning petabytes of data into millions in cost savings for the worlds biggest retailers

December 8, 2019

Jonathon Whitton details how PRGX is using Talend and Cloudera to load two million annual client flat files into a Hadoop cluster and perform recovery audit services in order to help clients detect, find, and fix leakage in their procurement and payment processes.
Benefits of scaling machine learning

Benefits of scaling machine learning

December 7, 2019

Machine learning is evolving to utilize new hardware, such as GPUs and large commodity clusters. Reza Zadeh presents two projects that have benefitted greatly through scaling: obtaining leading results on the Princeton ModelNet object recognition task and matrix computations and optimization in Apache Spark.
When one data center is not enough: Building large-scale stream infrastructures across multiple data centers with Apache Kafka

When one data center is not enough: Building large-scale stream infrastructures across multiple data centers with Apache Kafka

December 7, 2019

You may have successfully made the transition from single machines and one-off solutions to large, distributed stream infrastructures in your data center. But what if one data center is not enough? Ewen Cheslack-Postava explores resilient multi-data-center architecture with Apache Kafka, sharing best practices for data replication and mirroring as well as disaster scenarios and failure handling.
Consuming cloud services with the Kubernetes Service Catalog

Consuming cloud services with the Kubernetes Service Catalog

December 2, 2019

Neil Peterson leads a technical deep dive into using the Kubernetes Service Catalog to dynamically provision and consume managed cloud services.
Kubernetes bootcamp: Deploying and scaling microservices

Kubernetes bootcamp: Deploying and scaling microservices

December 1, 2019

Kubernetes has a reputation for being complex to set up and operate, but that doesn't have to be the case. Join Jrme Petazzoni to explore Kubernetes concepts and architecture and learn how to use it to deploy and scale your applications. The content is suitable to all kinds of deployment models, from the cloud (AKS, EKS, GKE, kops, etc.) to on-premises.
Kubernetes: Crossing the chasm

Kubernetes: Crossing the chasm

December 1, 2019

As Kubernetes enters the mainstream market, we are seeing more use cases that don't fit the original mold, each bringing a new set of challenges. Ian Crosby discusses three specific case studies, the challenges encountered adopting Kubernetes, and the solutions and tooling used to solve them.
Smooth scaling: Slacks journey toward a new database

Smooth scaling: Slacks journey toward a new database

November 29, 2019

Slacks rapid growth over the last few years outpaced the original databases scaling capacity, which negatively impacted the company's customers and engineers. Ameet Kotian explains how a small team of engineers embarked on a journey for the right database solution, which eventually led them to Vitess, an open source cluster database.
Switching horses midstream: The challenges of migrating 150+ microservices to Kubernetes

Switching horses midstream: The challenges of migrating 150+ microservices to Kubernetes

November 29, 2019

The Financial Times recently migrated its content platform to Kubernetes. Join Sarah Wells to find out what it takes to migrate 150+ microservices from one container stack to another without affecting the existing production users and while the rest of your teams are working on delivering new functionality.
The simply complex task of implementing Kubernetes ingress: Lessons learned

The simply complex task of implementing Kubernetes ingress: Lessons learned

November 28, 2019

Getting traffic into a Kubernetes cluster should be simple, but its not. The range of options can be confusing, and implementing effective configuration is equally challenging. Richard Li discusses the evolution of ingress on Kubernetes, explains why ingress controllers arent necessarily the best approach, and shares a series of lessons learned about managing traffic ingress.
Who guards the guardians? Designing for resilience in cluster orchestrators

Who guards the guardians? Designing for resilience in cluster orchestrators

November 28, 2019

Preetha Appan outlines various failure modes ranging from network failures to entire server failures in Nomad, an open source scheduler that supports heterogeneous workloads.
Apache Eagle: Secure Hadoop in real time

Apache Eagle: Secure Hadoop in real time

November 21, 2019

Apache Eagle is an open source monitoring solution to instantly identify access to sensitive data, recognize malicious activities, and take action. Arun Karthick Manoharan, Edward Zhang, and Chaitali Gupta explain how Eagle helps secure a Hadoop cluster using policy-based and machine-learning user-profile-based detection and alerting.
Breaking Spark: Top five mistakes to avoid when using Apache Spark in production

Breaking Spark: Top five mistakes to avoid when using Apache Spark in production

November 20, 2019

Spark has been growing in deployments for the past year. The increasing amount of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine. Drawing on his experiences across 150+ production deployments, Neelesh Srinivas Salian explores common issues observed in a cluster environment setup with Apache Spark.
HopsWorks: Multitenant Hadoop as a service

HopsWorks: Multitenant Hadoop as a service

November 18, 2019

Currently, multitenancy in Hadoop is limited to organizations running separate Hadoop clusters, and the secure sharing of resources is achieved using virtualization or containers. Jim Dowling describes how HopsWorks enables organizations to securely share a single Hadoop cluster using projects and a new metadata layer that enables protection domains while still allowing data sharing.
Introduction to generalized low-rank models and missing values

Introduction to generalized low-rank models and missing values

November 18, 2019

The generalized low-rank model is a new machine-learning approach for reconstructing missing values and identifying important features in heterogeneous data. Through a series of examples, Jo-fai Chow demonstrates how to fit low-rank models in a parallelized framework and how to use these models to make better predictions.
Stream analytics in the enterprise: A look at Intels internal IoT implementation

Stream analytics in the enterprise: A look at Intels internal IoT implementation

November 17, 2019

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was based on open source big data technologies and containers and was designed as a multitenant platform with built-in analytical capabilities. Moty highlights the key lessons learned from this journey and offers a thorough review of the platforms architecture.
Containers at scale thanks to Kubernetes

Containers at scale thanks to Kubernetes

November 13, 2019

Security, easy scaling, and high availability are essential needs for real-world application infrastructure. Brandon Philips offers a how-to talk and tutorial demonstrating how to set up a production-ready cluster with full TLS, manage the cluster through scaling and upgrade, and build the cluster for high availability. Brandon also addresses upgrading Kubernetes while keeping availability.
Deployment and orchestration at scale with Docker Swarm

Deployment and orchestration at scale with Docker Swarm

November 13, 2019

Jrme Petazzoni and AJ Bowen demonstrate building an app from development to production with Docker. Jrme and AJ run a sample app on a single node with Compose and add scaling and load balancing. They then provision a Swarm cluster with Docker Machine and implement multihost communication with overlay networking. The result will be a highly available, scalable deployment for the application.
Effective Docker Swarm

Effective Docker Swarm

November 12, 2019

SwarmDocker's answer to clusteringtreats a group of Docker hosts as a single host. Everett Toews explains how to Swarm with Carina, a service that builds Docker Swarm clusters for you.
Intro to Docker Swarm

Intro to Docker Swarm

November 10, 2019

Everett Toews offers an introduction to Docker Swarm and explains what it takes to develop and deploy an application on it.
Migrate your traditional VM-based clusters to containers

Migrate your traditional VM-based clusters to containers

November 8, 2019

Arun Gupta explains how to migrate a traditional VM-based cluster to a container-based infrastructure. You'll learn the gotchas of Docker Cloud, Kubernetes on Google/Amazon, OpenShift, and Mesos + Marathon.
Monitoring in motion: Monitoring containers and dynamic infrastructure

Monitoring in motion: Monitoring containers and dynamic infrastructure

November 8, 2019

We rely on our monitoring to tell us when our services, applications, or infrastructure diverge from "normal." Ilan Rabinovitch takes a deep dive into techniques for leveraging service discovery into your monitoring workflow and explains how to instrument your code in your containers and track the performance and availability of your applications as they move around.