Containers and anycast IPs at DigitalOcean

Andrew Kim leads a technical deep dive into how DigitalOcean uses anycast IPs, BGP, and Kubernetes to run globally distributed services on containers. Along the way, Andrew discusses design considerations for scalability, architectural trade-offs, data center networking, lessons learned in production, and challenges to adopting containers for latency sensitive applications.

Today’s container networking technology has made it significantly easier to build distributed systems on top of container orchestrators such as Kubernetes, Mesosphere, and Docker Swarm. Container networking technologies use Linux primitives such as iptables and IPVS to provide load-balancing capabilities for network traffic across containers in a cluster. These simple yet powerful tools are a cornerstone of successful containerized systems, as they provide highly available environments with little to no effort. Despite the many benefits of container networking, running containerized applications that must be latency sensitive and globally distributed is an extremely challenging task. Container networking is mainly scoped for in-cluster traffic, leaving little room to globally distribute an application across multiple clusters. Moreover, extending a container network for external traffic requires many additional layers of abstraction, usually introducing points of failures in a cluster and increasing end-to-end latency. Andrew Kim leads a technical deep dive into how DigitalOcean uses anycast IPs, BGP, and Kubernetes to run globally distributed services on containers. Along the way, Andrew discusses design considerations for scalability, architectural trade-offs, data center networking, lessons learned in production, and challenges to adopting containers for latency sensitive applications.