The golden idol swap: Pragmatic database migration

Switching databases requires a lot of effort from engineering teams, and Christian Grabowski walks you through steps you can take to reduce the amount of work needed to achieve payoff. NS1 created an abstraction layer of wire protocols between old and new databases, which allowed it to develop advanced functionality in new services, while legacy services required minimal changes.

Whether it’s to scale the throughput of your system, the amount of data you can store, or addressing technical or business issues, companies sometimes have to switch to a new database. There’s a significant amount of work that goes into planning and implementing this switch, including designing schemas, coming up with a failover plan, and figuring out how to distribute data across data centers. On top of all that, migrating legacy services and data to the new database is a large challenge. When NS1 set out to switch from a document store database to a relational database, it decided to abstract the databases through a proxy to minimize the amount of code changes required for its legacy services. The proxy provided wire protocol layer compatibility to minimize changes at legacy call sites. If NS1 didn’t go down this route, it would’ve had to swap out the database driver for several services, which were implemented in multiple languages. Additionally, the company decided to not use a protocol like HTTP or gRPC to accomplish this, as this would require modifying various function calls throughout its legacy codebase still. With those options not well suited to the company, it decided to design a proxy that would use the document store’s wire protocol in the frontend. Christian Grabowski describes how NS1 implemented this proxy, what hurdles it faced, and what tough design decisions it had to make along the way, as well as how it reduced the amount of development effort required to successfully make the switch. As for requirements for a backend of the proxy, the company needed to be able to still write to the legacy database until its data was entirely moved over to the new database, so the proxy was either required to write to both backends or just one based on configuration. NS1 also needed the proxy to route to this correct databases based on collection and table as it planned to run this proxy both in its core and edge facilities. But it wasn’t always smooth sailing, considering the significant differences between the two databases. NS1 had to make some sacrifices with regard to the features between the databases.