ETL and event sourcing

Have you ever launched a large ETL job to check a fix for a corner case in a derived calculation or normalization? Marc Siegel shares lessons learned applying the event sourcing pattern within an ETL pipeline. Key takeaway in regex form: E{1}TL – that is, "Extract once, transform and load infinite times."

Traditional ETL pipelines, consisting of extract, transform, and load stages, are a staple integration architecture pattern used in a wide variety of business domains. They present entire classes of familiar frustrations and impedance mismatches that many engineers have encountered firsthand. More recently the concept of a data lake has grown in popularity, bringing certain ideas from domain-driven design, such as bounded contexts, to bear on these problem domains. Can you go even further into the world of DDD? What costs and benefits would you observe if you did? Marc Siegel shares a real-life case study and lessons learned from going further into domain-driven design and applying the event sourcing pattern to the traditional problem domain of an ETL pipeline. If your work entails bringing lots of data into your system and building state off of it, you may find this talk interesting. Topics include: