Using Spark for crunching astronomical data on the LSST scale

The Large Scale Survey Telescope (LSST) is one of the most important future surveys. Its unique design allows it to cover large regions of the sky and obtain images of the faintest objects. After 10 years of operation, it will produce about 80 PB of data in images and catalog data. Petar Zecevic explains AXS, a system built for fast processing and cross-matching of survey catalog data.

The slew of upcoming large-scale astronomical surveys promises exciting times for astronomy and computer science. One of the most important future surveys is the LSST. Its unique design and excellent location allow it to go both wide and deep at the same time, covering large regions of the sky and obtaining images of the faintest objects. LSST will produce one 3.2 giga-pixel image every 20 seconds every night for 10 years, resulting in the first “video” of the deep sky in history and (according to some estimates) about 80 PB of data. Furthermore, the worldwide scientific community will receive real-time alerts triggered by changes in the sky within 60 seconds of their detection. Petar Zecevic explains how the LSST image processing pipeline uses acquired images to produce catalogs of astronomical objects. Together with colleagues from University of Washington, Petar built Astronomy Extensions for Spark (AXS), a system for processing and quickly cross-matching catalog data, based on Apache Spark. You’ll learn about its architecture and what’s behind its great performance.