Strata Data Conference

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x performance improvement to Qunars streaming processing

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x performance improvement to Qunars streaming processing

December 2, 2019

Alluxiothe first memory-speed virtual distributed storage system in the worldunifies the data from various under storage systems and presents a global namespace to various computation frameworks. Xueyan Li and Yupeng Fu explore how Alluxio has led to performance improvements averaging a 300x improvement at service peak time on stream processing workloads at Qunar.
Journey to AWS: Straddling two worlds

Journey to AWS: Straddling two worlds

December 2, 2019

As Intuit moves its SaaS platform from its own data centers to AWS, it will straddle both worlds for a period of time (and potentially indefinitely). Calum Murray looks at what straddling means to data and data systems.
Lessons from piloting the London Office of Data Analytics

Lessons from piloting the London Office of Data Analytics

December 2, 2019

Eddie Copeland shares lessons learned from piloting the London Office of Data Analytics, a collaboration between the Greater London Authority, Nesta, and ASI Data Science that is exploring the potential of applying data analytics to reform public services.
Lessons learned working with Spark and Cassandra

Lessons learned working with Spark and Cassandra

December 2, 2019

Matthias Niehoff shares lessons learned working with Spark, Cassandra, and the Spark-Cassandra connector and best practices drawn from his work on multiple big and fast data projects, as well as challenges encountered along the way.
Machine learning is a moonshot for us all (sponsored by Google)

Machine learning is a moonshot for us all (sponsored by Google)

December 2, 2019

Data analytics and machine learning are the drivers of the fourth industrial revolution. As technologists, we stand on the brink of incredible opportunity. Darren Strange explores the tremendous opportunity we have before us and asks, will we be pioneers creating new possibilities or will we hold on to the past?
Making recommendations using graphs and Spark

Making recommendations using graphs and Spark

December 2, 2019

Harry Powell and Raffael Strassnig demonstrate how to model unobserved customer preferences over businesses by thinking about transactional data as a bipartite graph and then computing a new similarity metricthe expected degrees of separationto characterize the full graph.
Meet the Expert with Phillip Radley (BT)

Meet the Expert with Phillip Radley (BT)

December 1, 2019

BT has adopted Hadoop as an enterprise platform for data processing and storage. Come talk to Phillip to find out how they did itand what you can learn from their experiences.
Meta-data science: When all the world's data scientists are just not enough

Meta-data science: When all the world's data scientists are just not enough

December 1, 2019

What if you had to build more models than there are data scientists in the worlda feat enterprise companies serving hundreds of thousands of businesses often have to do? Leah McGuire offers an overview of Salesforce's general-purpose machine-learning platform that automatically builds per-company optimized models for any given predictive problem at scale, beating out most hand-tuned models.
Migrating petabyte-scale Hadoop clusters with zero downtime

Migrating petabyte-scale Hadoop clusters with zero downtime

December 1, 2019

Migrating petabyte-scale Hadoop installations to a new cluster with hundreds of machines, several thousands of jobs daily, and countless ecosystem integrations while maintaining a stable production environment is a challenging task. Alon Elishkov discusses the techniques and tools Outbrain has developed to achieve this goal.
Multinode restricted Boltzmann machines for big data

Multinode restricted Boltzmann machines for big data

December 1, 2019

Nikolay Manchev offers an overview of the restricted Boltzmann machine, a type of neural network with a wide range of applications, and shares his experience using it on Hadoop (MapReduce and Spark) to process unstructured and semistructured data at a scale.
Near-real-time ingest with Apache Flume and Apache Kafka at 1 million-events-per-second scale

Near-real-time ingest with Apache Flume and Apache Kafka at 1 million-events-per-second scale

December 1, 2019

Vodafone UKs new SIEM system relies on Apache Flume and Apache Kafka to ingest over 1 million events per second. Tristan Stevens discusses the architecture, deployment, and performance-tuning techniques that enable the system to perform at IoT-scale on modest hardware and at a very low cost.
Organizing the data lake

Organizing the data lake

December 1, 2019

Building a data lake involves more than installing and using Hadoop. The goal in most organizations is to build multiuse data infrastructure that is not subject to past constraints. Mark Madsen discusses hidden design assumptions, reviews design principles to apply when building multiuse data infrastructure, and provides a reference architecture.
Peeking into the black box: Lessons from the front lines of machine-learning product launches

Peeking into the black box: Lessons from the front lines of machine-learning product launches

December 1, 2019

Grace Huang shares lessons learned from running and interpreting machine-learning experiments and outlines launch considerations that enable sustainable, long-term ecosystem health.
Performance and security: A tale of two cities

Performance and security: A tale of two cities

December 1, 2019

Performance and security are often at loggerheads. Rekha Joshi explains why and offers a deep dive into how performance and security are managed in some of the most intense and critical data platform services at Intuit.
Presto: Distributed SQL done faster

Presto: Distributed SQL done faster

December 1, 2019

Wojciech Biela and ukasz Osipiuk offer an introduction to Presto, an open source distributed analytical SQL engine that enables users to run interactive queries over their datasets stored in various data sources, and explore its applications in various big data problems.
Real-time intelligence gives Uber the edge

Real-time intelligence gives Uber the edge

December 1, 2019

M. C. Srivas covers the technologies underpinning the big data architecture at Uber and explores some of the real-time problems Uber needs to solve to make ride sharing as smooth and ubiquitous as running water, explaining how they are related to real-time big data analytics.
Real-time machine learning with Redis, Apache Spark, TensorFlow, and more

Real-time machine learning with Redis, Apache Spark, TensorFlow, and more

November 30, 2019

Kamran Yousaf explains how to substantially accelerate and radically simplify common practices in machine learning, such as running a trained model in production, to meet real-time expectations, using Redis modules that natively store and execute common models generated by Spark ML and TensorFlow algorithms.
Reliable prediction: Handling uncertainty

Reliable prediction: Handling uncertainty

November 30, 2019

Reliable prediction is the ability of a predictive model to explicitly measure the uncertainty involved in a prediction without feedback. Robin Senge shares two approaches to measure different types of uncertainty involved in a prediction.
Rethinking stream processing with Apache Kafka: Applications versus clusters and streams versus databases

Rethinking stream processing with Apache Kafka: Applications versus clusters and streams versus databases

November 30, 2019

Michael Noll explains how Apache Kafka helps you radically simplify your data processing architectures by building normal applications to serve your real-time processing needs rather than building clusters or similar special-purpose infrastructurewhile still benefiting from properties typically associated exclusively with cluster technologies.
Spark and R with sparklyr

Spark and R with sparklyr

November 30, 2019

R is a top contender for statistics and machine learning, but Spark has emerged as the leader for in-memory distributed data analysis. Douglas Ashton, Aimee Gott, and Mark Sellors introduce Spark, cover data manipulation with Spark as a backend to dplyr and machine learning via MLlib, and explore RStudio's sparklyr package, giving you the power of Spark without having to leave your R session.
Stream analytics with SQL on Apache Flink

Stream analytics with SQL on Apache Flink

November 30, 2019

Although the most widely used language for data analysis, SQL is only slowly being adopted by open source stream processors. One reason is that SQL's semantics and syntax were not designed with streaming data in mind. Fabian Hueske explores Apache Flink's two relational APIs for streaming analyticsstandard SQL and the LINQ-style Table APIdiscussing their semantics and showcasing their usage.
The business case for deep learning, Spark, and friends

The business case for deep learning, Spark, and friends

November 30, 2019

Deep learning is white-hot at the moment, but why does it matter? Developers are usually the first to understand why some technologies cause more excitement than others. Sanjay Mathur relates this insider knowledge, providing a tour through the hottest emerging data technologies of 2017 to explain why theyre exciting in terms of both new capabilities and the new economies they bring.
The data subject first?

The data subject first?

November 30, 2019

You may have heard that a piece of legislation called the GDPR is looming. Aurlie Pols draws a broad philosophical picture of the data ecosystem we are all a part of before honing into the right to data portability, hopefully empowering you to reclaim your data subject rights.
The digital twin: Real and gaining ground

The digital twin: Real and gaining ground

November 30, 2019

A digital twin is a virtual model of a product or service that allows analysis of data and monitoring of systems to avert problems before they even occurand even plan for the future by using simulations. Shree Dandekar explores a new cloud-based service from the Honeywell Connected Plant that provides industrial users with around-the-clock monitoring of plant data and rigorous simulations.
The science of visual interactions

The science of visual interactions

November 29, 2019

Miriam Redi explores the invisible side of visual data, investigating how machine learning can detect subjective properties of images and videos, such as beauty, creativity, sentiment, style, and more curious characteristics. Miriam shows how these detectors can be applied in the context of web media search, advertising, and social media.
The state of Spark in the cloud

The state of Spark in the cloud

November 29, 2019

Nicolas Poggi evaluates the out-of-the-box support for Spark and compares the offerings, reliability, scalability, and price-performance from major PaaS providers, including Azure HDinsight, Amazon Web Services EMR, Google Dataproc, and Rackspace Cloud Big Data, with an on-premises commodity cluster as baseline.
Transport for London: Using data to keep London moving

Transport for London: Using data to keep London moving

November 29, 2019

Transport for London (TfL) and WSO2 have been working together on broader integration projects focused on getting the most efficient use out of London transport. Roland Major and Sriskandarajah Suhothayan explain how TfL and WSO2 bring together a wide range of data from multiple disconnected systems to understand current and predicted transport network status.
Unified stateful big data processing in Apache Beam (incubating)

Unified stateful big data processing in Apache Beam (incubating)

November 29, 2019

Apache Beam's new State API brings scalability and consistency to fine-grained stateful processing while remaining portable to any Beam runner. Aljoscha Krettek introduces the new state and timer features in Beam and shows how to use them to express common real-world use cases in a backend-agnostic manner.