Architecting a data platform to support analytic workflows for scientific data

In upstream oil and gas, a vast amount of the data requested for analytics projects is scientific data: physical measurements about the real world. Historically, this data has been managed library style, but a new system was needed to best provide this data. Sun Maria Lehmann and Jane McConnell share architectural best practices learned from their work with subsurface data.

In upstream and oil and gas, by far the largest category of data by volume is scientific data about the subsurface, in the form of seismic surveys, petrophysical well logs, drilling logs, and laboratory data and report data from physical samples. Historically, this data has been managed for archive in library-style solutions, and subsets of the data were loaded to specialist applications to perform standard workflows. These applications limited the amount of data that could be analyzed together and restricted the types of analysis that could be done. This scientific data is perhaps the most important data to expose to new styles of analytics and machine learning in terms of the business value this can unlock, but to do it at scale, across all relevant data, and often in near real time, a new approach to the data architecture was required. Sun Maria Lehmann and Jane McConnell share architectural best practices learned from their work building a subsurface data platform in a cloud environment that is designed from the ground up to support analytical processing. Sun and Jane outline the key architectural patterns needed for managing physical measurement data, detail their approach to standardizing and integrating this data, and discuss the technology choices they made.