Strata + Hadoop World

Drawing insights from imperfection: A year of Dear Data

November 18, 2019

Stefanie Posavec recently completed a year-long drawing project with Giorgia Lupi called Dear Data, where each week they manually gathered and drew their data on a postcard to send to the other. Stefanie discusses the professional insights she gained spending a year on such an intensive personal data project.

HopsWorks: Multitenant Hadoop as a service

November 18, 2019

Currently, multitenancy in Hadoop is limited to organizations running separate Hadoop clusters, and the secure sharing of resources is achieved using virtualization or containers. Jim Dowling describes how HopsWorks enables organizations to securely share a single Hadoop cluster using projects and a new metadata layer that enables protection domains while still allowing data sharing.

How to date (a film)

November 18, 2019

Using data science to help make better business decisions in the film industry.

Introduction to generalized low-rank models and missing values

November 18, 2019

The generalized low-rank model is a new machine-learning approach for reconstructing missing values and identifying important features in heterogeneous data. Through a series of examples, Jo-fai Chow demonstrates how to fit low-rank models in a parallelized framework and how to use these models to make better predictions.

Machine learning for human rights advocacy: Big benefits, serious consequences

November 18, 2019

Megan Price demonstrates how machine-learning methods help us determine what we know, and what we don't, about the ongoing conflict in Syria. Megan then explains why these methods can be crucial to better understand patterns of violence, enabling better policy decisions, resource allocation, and ultimately, accountability and justice.

Making sense of exactly-once semantics

November 18, 2019

Exactly-once semantics is a highly desirable property for streaming analytics. Ideally, all applications process events once and never twice, but making such guarantees in general either induces significant overhead or introduces other inconveniences, such as stalling. Flavio Junqueira explores what's possible and reasonable for streaming analytics to achieve when targeting exactly-once semantics.

Modern data strategy and CERN

November 17, 2019

Clouderas Mike Olson is joined by Manuel Martin Marquez, a senior research and data scientist at CERN, to discuss Hadoop's role in the modern data strategy and how CERN is using Hadoop and information discovery to help drive operational efficiency for the Large Hadron Collider.

Petascale genomics

November 17, 2019

The advent of next-generation DNA sequencing technologies is revolutionizing life sciences research by routinely generating extremely large datasets. Tom White explains how big data tools developed to handle large-scale Internet data (like Hadoop) help scientists effectively manage this new scale of data and also enable addressing a host of questions that were previously out of reach.

Pixels and place: What online experiences can borrow from offline spaces and vice versa

November 17, 2019

The metaphors used online have always borrowed heavily from the offline world, but as our online and offline worlds converge, the biggest opportunities for innovative experiences will come from blending them intentionally. Kate ONeill examines how the meaning and understanding of place relates to identity, culture, and intent and how we can shape our audiences' experiences more meaningfully.

Prophecies and predictive models: How a 3D approach to data transforms your business

November 17, 2019

The famous Oracle at Delphi had a secret: Its prophecies were interpreted by Temple Guides, using a very early version of ethnographic research. With todays near-blind faith in the predictive power of Big Data, its time to take a lesson from the Ancient Greeks.

Protecting individual privacy in a data-driven world

November 17, 2019

With the analytic and predictive power of big data comes the responsibility to respect and protect individual privacy. As citizens, we should hold organizations to account; as data practitioners, we must find intelligent ways to analyze data without violating privacy. Jason McFall discusses privacy risks and surveys leading privacy-preserving analysis techniques.

Realizing the value of combining the IoT and big data analytics

November 17, 2019

The IoT combined with big data analytics enables organizations to track new patterns and signs and bring data together that previously was not only a challenge to integrate but also way too expensive. Frank Saeuberlich and Eliano Marques explain why data management, data integration, and multigenre analytics are foundational to driving business value from IoT initiatives.

Saving whales with deep learning

November 17, 2019

Piotr Niedwied explores how deepsense.io created the worlds best deep learning model for identifying individual right whales using aerial photography for the NOAA (National Oceanic and Atmospheric Administration) and explains what happened when the solution was covered by news media around the globe.

Sightseeing, venues, and friends: Predictive analytics with Spark ML and Cassandra

November 17, 2019

Which venues have similar visiting patterns? How can we detect when a user is on vacation? Can we predict which venues will be favorited by users by examining their friends' preferences? Natalino Busa explains how these predictive analytics tasks can be accomplished by using Spark SQL, Spark ML, and just a few lines of Scala code.

Simple, fast, and flexible risk aggregation in Hadoop

November 17, 2019

Value at risk (VaR) is a widely used risk measure. VaR is not simply additive, which provides unique challenges to report VaR at any aggregate level, as traditional database aggregation functions don't work. Deenar Toraskar explains how the Hive complex data types and user-defined functions can be used very effectively to provide simple, fast, and flexible VaR aggregation.

Stream analytics in the enterprise: A look at Intels internal IoT implementation

November 17, 2019

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was based on open source big data technologies and containers and was designed as a multitenant platform with built-in analytical capabilities. Moty highlights the key lessons learned from this journey and offers a thorough review of the platforms architecture.

TensorFlow: Machine learning for everyone

November 17, 2019

TensorFlow is an open source software library for numerical computation with a focus on machine learning. Its flexible architecture makes it great for research and production deployment. Sherry Moore offers a high-level introduction to TensorFlow and explains how to use it to train machine-learning models to make your next application smarter.

The business bottom line of data lakes: Real-life experiences

November 16, 2019

Data is no longer a by-product of business transactions; now, data is the business. Franz Aman explains how data lakes can put the power of big data into the hands of every business person, sharing the inside scoop on how he turned marketing into a new kind of revenue-generation machine and interviewing an Informatica customer about how data lakes have innovated and transformed their business.

The curious case of the data scientist

November 16, 2019

David Selby shares some of the challenges he has faced coercing meaning from data and explains why he is particularly enthusiastic about the latest technological developments in the data science field.

The future of (artificial) intelligence

November 16, 2019

The news media in recent months has been full of dire warnings about the risk that AI poses to the human race. Should we be concerned? If so, what can we do about it? While some in the mainstream AI community dismiss these concerns, Stuart Russell argues that a fundamental reorientation of the field is required.

The Internet of Things: Its the (sensor) data, stupid

November 16, 2019

Martin Willcox shares the lessons he's learned from successful Teradata IoT projects about about how to manage and leverage sensor data and explains why data management, data integration, and multigenre analytics are foundational to driving business value from IoT initiatives.

Triggers in Apache Beam (incubating): User-controlled balance of completeness, latency, and cost in streaming big data pipelines

November 16, 2019

Drawing on important real-world use cases, Kenneth Knowles delves into the details of the language- and runner-independent semantics developed for triggers in Apache Beam, demonstrating how the semantics support the use cases as well as all of the above variability in streaming systems. Kenneth then describes some of the particular implementations of those semantics in Google Cloud Dataflow.

Watermarks: Time and progress in streaming dataflow and beyond

November 16, 2019

Watermarks are a system for measuring progress and completeness in out-of-order stream processing systems and are used to emit correct results in a timely way. Given the trend toward out-of-order processing in current streaming systems, understanding watermarks is an increasingly important skill. Slava Chernyak explains watermarks and demonstrates how to apply them using real-world cases.

What does your smart car know about you?

November 16, 2019

In the last few years, auto makers and others have introduced devices to connect cars to the Internet and gather data about the vehicles activity, and auto insurers and local governments are just starting to require these devices. Charles Givre gives an overview of the security risks as well as the potential privacy invasions associated with this unique type of data collection.

Whats next for music services? The answer is in the data

November 16, 2019

Can our real-time distributed data systems help predict whether high-resolution audio is the future of digital music? What about content curation? Paul Shannon and Alan Hannaway explore the future of music services through data and explain why 7digital believes well-curated, high-resolution listening experiences are the future of digital music services.

Why is my Hadoop job slow?

November 15, 2019

Hadoop is used to run large-scale jobs over hundreds of machines. Considering the complexity of Hadoop jobs, it's no wonder that Hadoop jobs running slower than expected remains a perennial source of grief for developers. Bikas Saha draws on his experience debugging and analyzing Hadoop jobs to describe the approaches and tools that can solve this difficult problem.

10 concepts the enterprise decision maker needs to understand about Hadoop

October 29, 2019

Figuring out Hadoop is daunting. However, understanding a set of basic yet important principles is all you need to cut through the hype and make intelligent enterprise decisions. Donald Miner breaks down modern Hadoop into 10 important principles you need to know to understand what Hadoop is and how it is different from the old way of doing things.

A few things engineers can learn from designers

October 29, 2019

Big data is great for feeding ML algorithms, but you quickly face a bandwidth issue when interfacing with humans. The brain is a fantastic information-processing machine and has an unparalleled, innate ability to detect patterns. Sbastien Pierre explains what designers can teach engineers about creating new ways to make large volumes of data understandable at the human level.