large-scale

How Open Networking can Manage Large-Scale VPCs on Top of Clos Network Architecture

January 7, 2020

Modern Data Centers are based on Clos Network Architecture, which is extremely simple yet unfriendly to applications. It is common to build Software Defined Data Centers on top of it and use VPCs to h …

Large-scale Network Simulations in Kubernetes

January 7, 2020

Virtual network simulation has been one of the best ways to test new designs before implementing them in production. There are plenty of tools that can help build arbitrary network topologies but most …

Serverless for data and AI

January 6, 2020

What is serverless, and how can it be utilized for data analysis and AI? Avner Braverman outlines the benefits and limitations of serverless with respect to data transformation (ETL), AI inference and training, and real-time streaming. This is a technical talk, so expect demos and code.

Accelerate innovation in the enterprise with distributed ML and DL (sponsored by BlueData)

January 4, 2020

Nanda Vijaydev shares practical examples ofand lessons learned fromML/DL use cases in financial services, healthcare, and other industries. You'll learn how to quickly deploy containerized multinode environments for TensorFlow and other ML/DL tools in a multitenant architecture either on-premises, in the cloud, or in a hybrid environment.

nGraph: Unlocking next-generation performance with deep learning compilers

January 4, 2020

The rapid growth of deep learning in demanding large-scale real-world applications has led to a rapid increase in demand for high-performance training and inference solutions. Adam Straw, Adam Procter, and Robert Earhart offer a comprehensive overview of Intel's nGraph deep learning compiler.

Aggregating Mobile Operator Edge Compute Infrastructure with End to End Network Slicing

January 3, 2020

We believe that edge is the missing piece in a new value chain that expands the types of devices on mobile networks, secures IOT, enables next generation content (AR/MR), readies mobile infrastructure …

Deep learning for recommender systems, Or How to compare pears with apples

January 1, 2020

Recommender systems support decision making with personalized suggestions and have proven useful in ecommerce, entertainment, and social networks. Sparse data and linear models are a burden, but the application of deep learning sets new boundaries and offers remarkable results. Join Marcel Kurovski to explore a use case for vehicle recommendations at Germany's biggest online vehicle market.

One-click deployment for containerized ML and DL environments

December 29, 2019

Nanda Vijaydev explains how to spin up instant ML/DL environments using containersall while ensuring enterprise-grade security and performance. Find out how to provide your data science teams with on-demand access to the tools and data they need, whether on-premises or in the cloud.

VPP Accelerated High Performance & Scalable L3DSR L4 Load Balancer on Top Clos

December 28, 2019

Delivering fast packets is getting difficult under rapid increase of service endpoints at huge traffic large-scale data center.Especially, time to on-demand deployment of load balancing function neede …

Cruise Control: Effortless management of Kafka clusters

December 26, 2019

Adem Efe Gencer explains how LinkedIn alleviated the management overhead of large-scale Kafka clusters using Cruise Control.

Masquerading malicious DNS traffic

December 22, 2019

Malicious DNS traffic patterns are inconsistent and typically thwart anomaly detection. David Rodriguez explains how Cisco uses Apache Spark and Stripes Bayesian inference software, Rainier, to fit the underlying time series distribution for millions of domains and outlines techniques to identify artificial traffic volumes related to spam, malvertising, and botnets (masquerading traffic).

Running Apache Samza on Kubernetes

December 22, 2019

Apache Samza is a distributed stream processing framework that allows you to process and analyze your data in real-time. It has been widely used at Linkedin and other companies on a large scale. Recen …

Serverless for data and AI

December 21, 2019

Spark adaptive execution: Unleash the power of Spark SQL

December 21, 2019

Spark SQL is widely used, but it still suffers from stability and performance challenges in highly dynamic environments with large-scale data. Haifeng Chen shares a Spark adaptive execution engine built to address these challenges. It can handle task parallelism, join conversion, and data skew dynamically during runtime, guaranteeing the best plan is chosen using runtime statistics.

7 years of DDD: Tackling complexity in large-scale marketing systems

December 19, 2019

Vladik Khononov explains how he and his team embraced domain-driven design (DDD) at Plexop, a large-scale marketing system that spans over a dozen different business domains. Join in to learn how DDD allowed the team to manage business complexities, see what worked (and what didn't), and discover where they had to adapt the DDD methodology to fit the company's needs.

Anatomy of testing in production: A Netflix original case study

December 19, 2019

So you think you can test your complex distributed application effectively just using your test environment? At Netflix, automated testing of client and server applications runs at scale in production. It has quickly gone from low-volume manual mode to automated continuous and voluminous mode. Vasanth Asokan offers a study of such testing at scale that will inform your overall testing strategy.

Running Large-Scale Stateful Workloads On Kubernetes at Lyft

December 9, 2019

Along with core services, K8s at Lyft also forms the base to run a large variety of data processing stateful data processing jobs which includes Spark, Flink and other jobs via various ML and Data pro …

A Toolkit for Simulating Kubernetes Scheduling at Scale

December 1, 2019

As Kubernetes becomes the de facto standard for container orchestration, new scheduling algorithms and systems are made for different scenarios and workloads. Unfortunately, it is very time and cost c …

Flyte: Cloud Native Machine Learning & Data Processing Platform

November 29, 2019

Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc …

Serverless Platform for Large Scale Mini-Apps: From Knative to Production

November 28, 2019

Serverless architecture is getting increasingly popular. However, developers are still experiencing pain points that hold them back from using it in production, like portability, interoperability and …

Improving Performance of Deep Learning Workloads With Volcano

November 27, 2019

Baidu internally has improved the performance of large-scale deep learning workloads by using the Volcano project. The CRD-based computing resource model makes it possible to use resources more effici …

Kubernetizing Big Data and ML Workloads at Uber

November 25, 2019

Uber relies on Big Data and ML to make business critical decisions such as pricing, trip ETA, etc. Today, those workloads such as Hive and Spark are running on YARN. To save millions of dollars by eff …

Release the Kraken: Bring Sidecar Containers to Next Level

November 19, 2019

Sidecar containers are well accepted and widely used nowadays. Sidecars are coupled with normal containers by sharing the same lifecycle and provide accessory features. This is a good pattern to enabl …

Intro: Harbor

November 11, 2019

As container technology become widely adopted in the industry, how to manage containerized applications poses new challenges to platform engineers. One of the challenges is to securely and efficiently …

Es-operator: Building an Elasticsearch Operator From the Bottom Up

November 10, 2019

Operating stateful applications is a challenge and Elasticsearch clusters are no exception. At Zalando, the search infrastructure heavily rely on large scale Elasticsearch clusters consisting of 100s …

Strategies to "Kubernetify" Legacy Applications

November 1, 2019

Microservice-based architectures have seen large-scale adoption and have become the industry standard for developing cloud-native applications. This growth is supplemented by key technologies like Doc …

Large Scale Distributed Deep Learning with Kubernetes Operators

October 29, 2019

The focus of this talk is the usage of Kubernetes operators to manage and automate training process for machine learning tasks. Two open source Kubernetes operators, tf-operator and mpi-operator, will …

10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You

October 28, 2019

Kubernetes is a very powerful and complicated system, and many users dont understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and …