Building a Fault-Tolerant Custom Resources Controller on Kubernetes

CRD (custom resource definition) is widely used to extend the behavior of Kubernetes. As all other kubernetes resources have controllers, so do CRDs. It is important that the custom resources are mana …

CRD (custom resource definition) is widely used to extend the behavior of Kubernetes. As all other kubernetes resources have controllers, so do CRDs. It is important that the custom resources are managed reliably during any failures of the custom controller specific to any CRD, such that a consistent state is maintained across failures. This talk will share the experiences we’ve gained: how leader election is used to ensure liveness, design principle of namespaced CRDs vs. clustered CRDs, how to handle unexpected events from APIServers, and maintaining the running system.