There are challenges to architecting a solution that will allow for developers to stream data into Kafka and be able to manage dirty data which is always an issue in ETL pipelines. I'd like to share l …

Hadoop-based data platforms that power ETL jobs and machine learning pipelines are great examples of monolithic architectures that could be redesigned with microservices. Stepan Pushkarev walks you through building and deploying data processing, reporting services, training, and prediction pipelines as decoupled microservices connected with the rest of the enterprise architecture.

Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? With Apache Kafka at the core, streaming platforms offer an entirely new perspective on managing the flow of data. Neha Narkhede shares examples of Kafka in action and explains why streaming platforms have become the central nervous system for modern digital businesses.

Neelesh Srinivas Salian offers an overview of the data platform used by data scientists at Stitch Fix, based on the Spark ecosystem. Neelesh explains the development process and shares some lessons learned along the way.

Common ETL jobs used for importing log data into Hadoop clusters require a considerable amount of resources, which varies based on the input size. Thiruvalluvan M G shares a set of techniquesinvolving an innovative use of Spark processing and exploiting features of Hadoop file formatsthat not only make these jobs much more efficient but also work well with fixed amounts of resources.

Avan Mathur shares strategies for database deployments and rollbacks as well as some patterns and best practices for reliably deploying databases as part of your CD pipeline, safely rolling back database code, ensuring data integrity, and more.