How Shutterfly migrated 10+ billion photos to the cloud

Jack Chan describes how Shutterfly migrated metadata from over 10B photos from a private data center into AWS in 100 days and explores designs to absorb mountains of metadata, on-premises ecommerce integration, and parallel user experiences, all in a highly scalable fashion. Shutterfly Photos is now a hybrid cloud solution with images hosted on-premises and client-facing photos metadata on AWS.

As part of Shutterfly’s multiyear Shutterfly 3.0 initiative, the company is modernizing its photo management solution. In 2016, the company took the bold step of migrating metadata from over 10 billion sets of photos from a private data center to AWS’s public cloud. This major feat was performed over the public internet across the North America continent, with a majority of those moved in less than 100 days—and within budget. Shutterfly Photos is now part of a hybrid cloud solution with image binaries hosted on-premises and metadata for photo management on AWS. Jack Chan focuses on how Shutterfly evolved a multiclient photo management system integrated through a prior acquisition and scale it to handle 20x the amount of data, outlining how the company transferred the metadata from an on-premises facility over the wire to AWS Cloud, persists it in a database cluster, postprocesses the photos with machine-learning algorithms, precaches the metadata for performance, hosts a scalable API, integrated its mobile, desktop, and web clients, added tons of monitors, and set up KPI metrics, all in time to prepare for Shutterfly’s Q4 holiday season sales. Jack relates the trials and tribulations of facing a huge unknown of taking on such a major initiative, the rally cry from technology leaders, and the mountains of scalability preparations that went on postmigration while running a parallel user experience within the tens of millions of paying customers.