cluster

Unraveling data with Spark using deep learning and other algorithms from machine learning

Unraveling data with Spark using deep learning and other algorithms from machine learning

December 28, 2019

Vartika Singh and Jeffrey Shmain walk you through various approaches using the machine learning algorithms available in Spark ML to understand and decipher meaningful patterns in real-world data. Vartika and Jeff also demonstrate how to leverage open source deep learning frameworks to run classification problems on image and text datasets leveraging Spark.
Enterprise Grade Security and Compliance Using Apache Atlas and Apache Ranger

Enterprise Grade Security and Compliance Using Apache Atlas and Apache Ranger

December 26, 2019

With an ever increasing need to secure and limit access to sensitive data, enterprises today need an open source solution. Apache Atlas - which is the metadata and governance framework for Hadoop join …
Using BPF in Kubernetes

Using BPF in Kubernetes

December 25, 2019

In this talk, I will present different use cases for using BPF in a Kubernetes cluster. BPF is a Linux in-kernel virtual machine and there are different kinds of BPF programs for different subsystems …
Ten Layers of Container Security

Ten Layers of Container Security

December 24, 2019

In this session, we'll identify the 10 most common layers in a typical container deployment, and the best ways to build security into each layer. These layers include: 1) container host: Multi-tenancy …
Using Docker Containers to Serve Deep Learing Predictions at Booking.com

Using Docker Containers to Serve Deep Learing Predictions at Booking.com

December 24, 2019

Each day, over 1.2 million room nights are reserved on Booking.com. That gives us access to huge amount of data which we can utilise in order to provide a better experience to our customers. We under …
Collaboration and automated operation as literate computing for reproducible infrastructure

Collaboration and automated operation as literate computing for reproducible infrastructure

December 23, 2019

Jupyter is useful for DevOps. It enables collaboration between experts and novices to accumulate infrastructure knowledge, while automation via notebooks enhances traceability and reproducibility. Yoshi Nobu Masatani shows how to combine Jupyter with Ansible for reproducible infrastructure and explores knowledge, workflow, and customer support as literate computing practices.
Labz 'N Da Wild 2.0: Teaching signal and data processing at scale using Jupyter notebooks in the cloud

Labz 'N Da Wild 2.0: Teaching signal and data processing at scale using Jupyter notebooks in the cloud

December 22, 2019

Demba Ba discusses two new signal processing/statistical modeling courses he designed and implemented at Harvard, exploring his perspective as an educator and that of the students as well as the steps that led him to adopt the current cloudJHub architecture. Along the way, Demba outlines the potential of architectures such as cloudJHub to help to democratize data science education.
Integrating deep learning libraries with Apache Spark

Integrating deep learning libraries with Apache Spark

December 18, 2019

Joseph Bradley and Xiangrui Meng share best practices for integrating popular deep learning libraries with Apache Spark, covering cluster setup, data ingest, configuring clusters, and monitoring jobs. Joseph and Xiangrui then demonstrate these techniques using Googles TensorFlow library.
A hands-on data science crash course for modeling and predicting the behavior of (large) distributed systems

A hands-on data science crash course for modeling and predicting the behavior of (large) distributed systems

December 17, 2019

Data science is a hot topic. Bart De Vylder offers a practical introduction that goes beyond the hype, exploring data analysis, visualization, and machine-learning techniques using Python for modeling the behavior of distributed systems. You'll leave with a solid starting point to implement data science techniques in your infrastructure or domain of interest.
Tutorial: Container Orchestration with Kubernetes

Tutorial: Container Orchestration with Kubernetes

December 17, 2019

This workshop introduces container orchestration with Kubernetes. Starting with a plain Linux systems, attendees learn how to create a on-premise Kubernetes cluster. Once having learned how to start a …
Canary in a coal mine: Building infrastructure resiliency with canary data reloads

Canary in a coal mine: Building infrastructure resiliency with canary data reloads

December 16, 2019

Remember the old practice of the canary in the coal mine, where miners used fragile feathered friends as a failure detector for toxic gasses? In software, a canary run is a trial executed on one machine before the rest of the cluster runs. Ann Kilzer explains how Indeed created a canary service leveraging Consuls key value store to improve the resilience of data reloads in any infrastructure.
How Shutterfly migrated 10+ billion photos to the cloud

How Shutterfly migrated 10+ billion photos to the cloud

December 15, 2019

Jack Chan describes how Shutterfly migrated metadata from over 10B photos from a private data center into AWS in 100 days and explores designs to absorb mountains of metadata, on-premises ecommerce integration, and parallel user experiences, all in a highly scalable fashion. Shutterfly Photos is now a hybrid cloud solution with images hosted on-premises and client-facing photos metadata on AWS.
Lightning Talk: How Kubernetes is Helpful for Accelerating Machine Learning Research and Engineering [I]

Lightning Talk: How Kubernetes is Helpful for Accelerating Machine Learning Research and Engineering [I]

December 15, 2019

In this lightning talk, the presenter shares his experience on helping machine learning research and engineering with kubernetes. k8s is not only a tool for managing microservices but also helpful for …
Lightning Talk: Telepresence: Local Development & Debugging of Remote Kubernetes Services

Lightning Talk: Telepresence: Local Development & Debugging of Remote Kubernetes Services

December 15, 2019

Developers who use Kubernetes for multi-container applications face a conundrum: develop locally or on a remote Kubernetes cluster. Local development adds complexity to your development environment, s …
Lightning Talk: CRI Proxy: Solving the Chicken-and-Egg Problem of Running a CRI Implementation as a DaemonSet [I]

Lightning Talk: CRI Proxy: Solving the Chicken-and-Egg Problem of Running a CRI Implementation as a DaemonSet [I]

December 14, 2019

CRI allows for special-purpose CRI implementations such as Virtlet, which makes it possible to run VMs as if they were containers. Still, deployment of these CRI implementations may bring us back to p …
Lightning Talk: Minikube Developer Workflow and Advanced Tips [B]

Lightning Talk: Minikube Developer Workflow and Advanced Tips [B]

December 14, 2019

A brief overview of the tools available in minikube to simplify building and testing your applications on a local Kubernetes cluster. - Bootstrapping minikube with kubeadm, - Running minikube in Tra …
Using Containers for Continuous Integration and Continuous Delivery [I]

Using Containers for Continuous Integration and Continuous Delivery [I]

December 14, 2019

Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but it increases complexity when scaling to multiple nodes and clusters. Jenkins is an example …
Establishing Container Trust at Scale [I]

Establishing Container Trust at Scale [I]

December 13, 2019

Quantifying risks in a container image is a critical aspect of production deployments. With orchestration clusters supporting thousands of nodes, any risk assessment solution must work at production s …
Prometheus Salon

Prometheus Salon

December 13, 2019

The Prometheus Salon will feature talks from Prometheus developers, including an introduction to Prometheus for beginners, a closer look at how you can use Prometheus to monitor your Kubernetes cluste …
Scaling HBase for big data (sponsored by Salesforce)

Scaling HBase for big data (sponsored by Salesforce)

December 13, 2019

Even though HBase is considered a highly scalable distributed solution, there are cases where the schema design of HBase tables or the way a client uses an HBase cluster may impact the scalability factor of HBase. Ranjeeth Karthik Selvan Kathiresan and Gurpreet Multani outline the most important things to consider when scaling your HBase cluster to accommodate high-volume and high-velocity data.
Distributed Workflows for Microservices-Style Applications [I] - Yun Qin, Nirmata

Distributed Workflows for Microservices-Style Applications [I] - Yun Qin, Nirmata

December 12, 2019

Microservices-style architectures solve several problems but also introduce new complexities. With Microservices, a best practice is to keep services isolated and loosely coupled. However, in the real …
SIG Cluster Lifecycle Update

SIG Cluster Lifecycle Update

December 12, 2019

The Cluster Lifecycle SIG is the Special Interest Group that is responsible for building the user experience for deploying and upgrading Kubernetes clusters. Since splitting out of SIG Cluster Ops in …
WG Container Identity Update

WG Container Identity Update

December 12, 2019

Decomposing applications into containers and microservices has many advantages but it creates a foundational problem: we need a reliable and secure way to identify all of the pieces. Kubernetes runs y …
Building a Secure, Multi-Protocol and Multi-Tenant Cluster for Internet-Facing Services [A]

Building a Secure, Multi-Protocol and Multi-Tenant Cluster for Internet-Facing Services [A]

December 10, 2019

Exposing internal HTTP-based services to the Internet is a well supported and documented feature of Kubernetes. What's less well understood is how to do it for thousands of services running on behalf …
CoreDNS Salon

CoreDNS Salon

December 10, 2019

Join us for an open discussion on CoreDNS! This will be an opportunity to learn more about CoreDNS, as well as discuss use cases, issues, and other matters with some of the maintainers. Well start wi …
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications to Kubernetes [B]

The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications to Kubernetes [B]

December 10, 2019

Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native ap …
BoF: Grafeas: Using Artifact Metadata to Audit, Govern, and Secure Your Software Supply Chain

BoF: Grafeas: Using Artifact Metadata to Audit, Govern, and Secure Your Software Supply Chain

December 9, 2019

Building software at scale requires strong governance of the software supply chain, and strong governance requires good data. This BoF will be a discussion around the recently launched Grafeas ("scrib …
Building a Cluster Management API using Kubicorn [A]

Building a Cluster Management API using Kubicorn [A]

December 9, 2019

Kris Nova (Heptio) and Robert Bailey (Google) join forces and begin the difficult task of looking into the future of the infrastructure layer of Kubernetes. We start the talk with a brief summary of t …