Lessons from an AWS migration

if(we)'s batch event processing pipeline is different from yours, but the process of migrating it from running in a data center to running in AWS is likely pretty similar. Chris Mills explains what was easier than expected, what was harder, and what the company wished it had known before starting the migration.

if(we) recently migrated all post-Kafka processing to AWS over about six months. The expected benefits were real. In particular, being able to provision stand-alone, batch-specific, spot instance-based EMR clusters in parallel has been a game changer. However, some aspects of the migration presented challenges. Parts of the stack expected the stability and consistency of our own racks and switches. A reliance on APIs from older versions of key tools, for instance, increased the integration timeline significantly. And the initial notion of automated deployment was significantly less than was needed. if(we)’s batch event processing pipeline is different from yours, but the process of migrating it from running in a data center to running in AWS is likely pretty similar. Chris Mills explains what was easier than expected, what was harder, and what the company wished it had known before starting the migration.