20 percent blissful, 80 percent ignorance

Data is all sales and marketing. The reality of data work is pain. Most data projects fail and are horrible experiences to work on. Phil Harvey explains that data is just too hardthe world needs to talk about real challenges so that we can start tackling them to deliver data projects that work. This is DataOps; there will be tears before bedtime.

There is a dirty little secret in all data work that everyone wants to ignore. It’s the 80 percent that has been called out in recent articles, under titles as diverse as “janitorial work” and “data engineering.” It’s real, it’s costly, and it hurts. It hurts business, and it hurts the people doing it. But some of us love it. It presents real challenges in data and makes everything work. Data is all sales and marketing. The reality of data work is pain. Most data projects fail and are horrible experiences to work on. Phil Harvey explains that data is just too hard—the world needs to talk about real challenges so that we can start tackling them to deliver data projects that work. This is DataOps; there will be tears before bedtime. When you break through the marketing and glamour of emerging data technologies, you fall into a pile of lies and difficult truths. You very quickly find battle lines drawn in a war of paradigms and fashion. The reality is that getting data is hard; talking to people about data is hard; robustness is hard; and the DataOps part of any “success” (few and far between as these are) is hardly every recognized. But there are things we can all do. Phil explores tools for understanding and communicating about the beautiful DataOps mess.