Leveraging metadata for automating delivery and operations of advanced data platforms

Peter Billen explains how to use metadata to automate delivery and operations of a data platform. By injecting automation into the delivery processes, you shorten the time to market while improving the quality of the initial user experience. Typical examples include data profiling and prototyping, test automation, continuous delivery and deployment, and automated code creation.

The importance of data has been growing over the past years, and companies are investing heavily in upgrading their data architectures and platforms. Driven by the democratization of technology, new opportunities in delivering value are becoming possible. Yet moving from a first project or proof of concept to an integrated solution as part of the ecosystem seems most of times to be a struggle. Even more, the fast evolution of the technology that originally led to implementing new solutions is creating even bigger challenges. Continuously new technologies are popping up, and they bring newer capabilities. The delivery processes are taken longer than foreseen, and the choices made previously are being scrutinized—or even worse, bypassed. At the same time, the request for perfection from the start is driving more complex solutions, even in organizations that have adopted agile ways of working. The key is being able to define a sufficiently long horizon and work toward a vision that defines a reference data architecture that suits the company. But even today, there’s a tendency to transpose an old way of working on top of a new technology platform. And it is exactly this constraint that holds back the road toward the data platform of the future. We need to keep an open mind about the path for growth, maturity, and evolution at the same time. We need to build a product, not deliver a project. And modularity is key here, because we want to avoid at all cost building today the legacy of tomorrow. A key aspect to highlight is the importance of the role that metadata plays here. Metadata—data describing the definition, context, classification, etc. of data itself—is growing in importance. Where previously this was considered more a by-product of technical components, it’s now the driver for operations performed by these same technical components. Different types of metadata can be combined and are able to deliver more insights and drivers for new services. For instance, when considering a simple text message, the value of the content is traditionally recognized, but knowing where it was sent, and when, and using which mobile phone, offers a new set of insights to be leveraged for creating value-added services. Peter Billen explains how Accenture takes this one step further, leveraging metadata to automate the delivery and operation of the data platform. By injecting automation into the delivery processes, you can shorten the time to market while improving the quality of the initial user experience. Typical examples include data profiling and prototyping, test automation, continuous delivery and deployment, and automated code creation. More than focusing on reducing the workload of the delivery teams, metadata automation can be used as part of the operations running inside the platform to automate data loads, classify data for trust and certification levels, integrate data access with data governance procedures, and dynamically expose data for API consumption.