How Netflix gives all its engineers SSH access to instances running in production

Traditional security methods can focus on putting barriers between people and resources, but sometimes the fastest way to solve a problem is to get shell access in production. Russell Lewis explains how Netflix decreased developer friction by building a certificate authority-based SSH bastion solution that balances security and engineering velocity needs.

One of the ways Netflix enables engineering velocity is with a culture of “freedom and responsibility” that empowers individuals with the freedom to do what is needed to get the job done. As a result, the security teams at Netflix focus on reducing developer friction, making it hard to do the wrong thing, and then rely on auditing, automated analysis, and alerting to keep things safe. Russell Lewis reviews a few approaches used in the industry to secure SSH bastions (aka jumpboxes) and evaluates them through the lens of Netflix’s security culture. Using these industry norms as the backdrop, Russell explains why Netflix decided it needed to build something new to enhance SSH bastion security. It needed something that was low friction for engineers but would allow for additional security features to be added in behind the scenes. Russell outlines Netflix’s SSH bastion architecture, which at its core uses SSO to authenticate engineers and then issues per-user credentials with short-lived certificates for SSH authentication of the bastion to an instance, which reduces the risk that they will be lost. Russell explores how this approach allows Netflix to audit and automatically alert after the fact, instead of slowing down engineers before granting access. Russell then presents the SSH certificate authority at the core of this system. It runs as an Amazon Web Services Lambda function and protects its private key with AWS’s Key Management Service. By relying only on AWS services, the SSH certificate authority is easy to bring up and can be used to bootstrap Netflix’s cloud deployments without adding circular dependencies. Netflix will also be announcing the open sourcing of BLESS (Bastion’s Lambda ephemeral SSH service).