big data

From BI to big data; Or, There and back again

From BI to big data; Or, There and back again

January 10, 2020

Francesco Mucio shares the basic tools he and his team had to learn (or relearn) moving from the coziness of their database to the big world of Spark, cloud, distributed systems, and continuous applications. It was an unexpected journey that ended exactly where it started: with an SQL query.
Getting ready for GDPR and CCPA: Securing and governing hybrid, cloud, and on-premises big data deployments

Getting ready for GDPR and CCPA: Securing and governing hybrid, cloud, and on-premises big data deployments

January 10, 2020

New regulations such as CCPA and GDPR are driving new compliance, governance, and security challenges for big data. Infosec and security groups must ensure a consistently secured and governed environment across multiple workloads. Mark Donsky, Ifigeneia Derekli, Lars George, and Michael Ernest share hands-on best practices for meeting these challenges, with special attention paid to CCPA.
How to mitigate mobile fraud risk by data analytics

How to mitigate mobile fraud risk by data analytics

January 9, 2020

Seonmin Kim offers an introduction to activities that mitigate the risk of mobile payments through various data analytical skills, drawn from actual case studies of mobile frauds, along with tree-based machine learning, graph analytics, and statistical approaches.
LSTM-based time series anomaly detection using Analytics Zoo for Spark and BigDL

LSTM-based time series anomaly detection using Analytics Zoo for Spark and BigDL

January 8, 2020

Collecting and processing massive time series data (e.g., logs, sensor readings, etc.) and detecting the anomalies in real time is critical for many emerging smart systems, such as industrial, manufacturing, AIOps, and the IoT. Guoqiong Song explains how to detect anomalies in time series data using Analytics Zoo and BigDL at scale on a standard Spark cluster.
Making the future

Making the future

January 8, 2020

James Burke asks whether we can use big data and predictive analytics at the social level to take the guesswork out of prediction and make the future what we all want it to be. If so, this would give us the tools to handle what looks like being the greatest change to the way we live since we left the caves.
Mutant tests too: The SQL

Mutant tests too: The SQL

January 8, 2020

Elliot West and Jay Green share approaches for applying software engineering best practices to SQL-based data applications to improve maintainability and data quality. Using open source tools, Elliot and Jay show how to build effective test suites for Apache Hive code bases and offer an overview of Mutant Swarm, a tool to identify weaknesses in tests and to measure SQL code coverage.
Lightning Talk: Telemetry Aware Scheduling: Let the (Scheduling) Magic Unfold!!

Lightning Talk: Telemetry Aware Scheduling: Let the (Scheduling) Magic Unfold!!

January 7, 2020

With containers becoming the desired deployment model for a wide range of workloads such as Big Data, IoT, storage, AI, ML, SDN and NFV solutions, Container orchestration tools like Kubernetes, need t …
Reinforcement learning: A gentle introduction and an industrial application

Reinforcement learning: A gentle introduction and an industrial application

January 6, 2020

Reinforcement learning (RL) learns complex processes autonomously like walking, beating the world champion in Go, or flying a helicopter. No big datasets with the right answers are needed: the algorithms learn by experimenting. Christian Hidber shows how and why RL works and demonstrates how to apply it to an industrial hydraulics application with 7,000 clients in 42 countries.
SDN in the World of Artificial Intelligence & Machine Learning

SDN in the World of Artificial Intelligence & Machine Learning

January 6, 2020

The promise of SDN has always been applications that can communicate behaviors and needed resources with the SDN controller via a well defined API. While traditional methods to determine the needed be …
Stream, stream, stream: Different streaming methods with Spark and Kafka

Stream, stream, stream: Different streaming methods with Spark and Kafka

January 6, 2020

NMC (Nielsen Marketing Cloud) provides customers (both marketers and publishers) with real-time analytics tools to profile their target audiences. To achieve that, the company needs to ingest billions of events per day into its big data stores in a scalable, cost-efficient way. Itai Yaffe explains how NMC continuously transforms its data infrastructure to support these goals.
The enterprise data cloud

The enterprise data cloud

January 5, 2020

The last decade has seen incredible changes in our technology. The advent of big data and powerful new analytic techniques, including machine learning and AI, means that we understand the world in ways that were simply impossible before. The simultaneous explosion of public cloud services has fundamentally changed our expectations of technology: it should be fast, simple, and flexible to use.
The vegan data diet: How Wikipedia cuts down privacy issues while keeping data fit

The vegan data diet: How Wikipedia cuts down privacy issues while keeping data fit

January 4, 2020

Analysts and researchers studying Wikipedia are hungry for long-term data to build experiments and feed data-driven decisions. But Wikipedia has a strict privacy policy that prevents storing privacy-sensitive data over 90 days. Marcel Ruiz Forns explains how the Wikimedia Foundation's analytics team is working on a vegan data diet to satisfy both.
There's something about data

There's something about data

January 4, 2020

Martin Leijen discusses how Rabobank created a data and intelligence lab as an enabler for data and business domains to accelerate in using AI and Advanced Analytics.
Executive Briefing: 5 key questions to kick off your AI implementation

Executive Briefing: 5 key questions to kick off your AI implementation

January 1, 2020

The significant hype bubble building up around AI has convinced many executives that if theyre not already tech savvy, they might not be ready for AIs transformative power. However, the reality is that AI is just another tool that can help your business, and youre probably not that far behind. Vinay Seth Mohta explains how to evaluate AI as you would any other strategic investment.
Executive Briefing: Overview of data governance

Executive Briefing: Overview of data governance

December 31, 2019

Effective data governance is foundational for AI adoption in enterprise, but it's an almost overwhelming topic. Paco Nathan offers an overview of its history, themes, tools, process, standards, and more. Join in to learn what impact machine learning has on data governance and vice versa.
Turn devices into data scientistsat the edge

Turn devices into data scientistsat the edge

December 28, 2019

Todays approach to processing streaming data is based on legacy big-data centric architectures, the cloud, and the assumption that organizations have access to data scientists to make sense of it allleaving organizations increasingly overwhelmed. Simon Crosby shares a new architecture for edge intelligence that turns this thinking on its head.
Analytics Zoo: Distributed TensorFlow and Keras on Apache Spark

Analytics Zoo: Distributed TensorFlow and Keras on Apache Spark

December 27, 2019

Jason Dai, Yuhao Yang, Jennie Wang, and Guoqiong Song explain how to build and productionize deep learning applications for big data with Analytics Zooa unified analytics and AI platform that seamlessly unites Spark, TensorFlow, Keras, and BigDL programs into an integrated pipelineusing real-world use cases from JD.com, MLSListings, the World Bank, Baosight, and Midea/KUKA.
Analytics Zoo: Distributed TensorFlow in production on Apache Spark

Analytics Zoo: Distributed TensorFlow in production on Apache Spark

December 27, 2019

Yuhao Yang and Jennie Wang demonstrate how to run distributed TensorFlow on Apache Spark with the open source software package Analytics Zoo. Compared to other solutions, Analytics Zoo is built for production environments and encourages more industry users to run deep learning applications with the big data ecosystems.
Augmented OLAP for big data from on-premises to multicloud (sponsored by Kyligence)

Augmented OLAP for big data from on-premises to multicloud (sponsored by Kyligence)

December 27, 2019

Augmenting data management and analytics platforms with artificial intelligence and machine learning is game changing for analysts, engineers, and other users. It enables companies to optimize their storage, speed, and spending. Yang Li details the Kyligence platform, which is evolving to the next level with augmented capabilities such as intelligent modeling, smart pushdowns, and more.
Executive Briefing: Big data in the era of heavy worldwide privacy regulations

Executive Briefing: Big data in the era of heavy worldwide privacy regulations

December 25, 2019

The implications of new privacy regulations for data management and analytics, such as the General Data Protection Regulation (GDPR) and the upcoming California Consumer Protection Act (CCPA), can seem complex. Mark Donsky and Nikki Rouda highlight aspects of the rules and outline the approaches that will assist with compliance.
Executive Briefing: Overview of data governance

Executive Briefing: Overview of data governance

December 25, 2019

Effective data governance is foundational for AI adoption in enterprise, but it's an almost overwhelming topic. Paco Nathan offers an overview of its history, themes, tools, process, standards, and more. Join in to learn what impact machine learning has on data governance and vice versa.
From flat files to deconstructed databases: The evolution and future of the big data ecosystem

From flat files to deconstructed databases: The evolution and future of the big data ecosystem

December 24, 2019

Big data infrastructure has evolved from flat files in a distributed filesystem to an efficient ecosystem to a fully deconstructed and open source database with reusable components. Julien Le Dem discusses the key open source components of the big data ecosystem and explains how they relate to each other and how they make the ecosystem more of a database and less of a filesystem.
How to protect big data in a containerized environment

How to protect big data in a containerized environment

December 23, 2019

Recent headline-grabbing data breaches demonstrate that protecting data is essential for every enterprise. The best-of-breed approach for big data is HDFS configured with Transparent Data Encryption (TDE). But TDE is difficult to configure and manageparticularly when run in Docker containers. Thomas Phelan discusses these challenges and explains how to overcome them.
ROCKSET: The design and implementation of a data system for low-latency queries for search and analytics

ROCKSET: The design and implementation of a data system for low-latency queries for search and analytics

December 22, 2019

Most existing big data systems prefer sequential scans for processing queries. Igor Canadi and Dhruba Borthakur challenge this view, offering an overview of converged indexing: a single system called ROCKSET that builds inverted, columnar, and document indices. Converged indexing is economically feasible due to the elasticity of cloud-resources and write optimized storage engines.
Running Apache Samza on Kubernetes

Running Apache Samza on Kubernetes

December 22, 2019

Apache Samza is a distributed stream processing framework that allows you to process and analyze your data in real-time. It has been widely used at Linkedin and other companies on a large scale. Recen …
Scaling visualization for big data and analytics in the cloud

Scaling visualization for big data and analytics in the cloud

December 21, 2019

Jaipaul Agonus and Daniel Monteiro do Carmo Rosa detail big data analytics and visualization practices and tools used by FINRA to support machine learning and other surveillance activities that the Market Regulation Department conducts in the AWS cloud.
Spark-PMoF: Accelerating big data analytics with Persistent Memory over Fabric

Spark-PMoF: Accelerating big data analytics with Persistent Memory over Fabric

December 21, 2019

Yuan Zhou, Haodong Tang, and Jian Zhang offer an overview of Spark-PMOF and explain how it improves Spark analytics performance.
The enterprise data cloud

The enterprise data cloud

December 20, 2019

Most enterprises want the same flexibility and convenience they get in the public cloud, no matter where their data lives or their applications run. We've reached the point that the "enterprise data cloud" must span the firewall and the services offered by hyperscale vendors. Mike Olson describes the key capabilities that such a system requires and why hybrid and multicloud is the future.