From intent to enforcement: Lessons from operating Kubernetes controllers at scale

Kubernetes controllers are what make the platform’s declarative model real. They observe the state, reconcile toward the intent, and keep doing so as the system changes beneath them. At larger scales, though, the hardest problems are no longer about writing the reconcile loop. They are about preserving correctness while caches lag, objects churn, and enforcement depends on a complete view of the world. This post distills what we learned from operating two critical controllers in Amazon EKS at scale. Kubernetes provides pods with connectivity, but it leaves fine-grained traffic enforcement to the underlying platform. Inside the cluster, intent such as who should be allowed to talk to whom still has to be translated into rules the datapath can enforce. For a small experiment, openness is convenient. For production traffic, it becomes a problem you need to solve, and you want to express that intent once and have it hold as pods are created and destroyed all day long. In Amazon EKS, two controllers close that gap. The Network Policy Controller governs traffic inside the cluster, deciding which pods may talk to which. The VPC Resource Controller governs how selected pods access AWS resources outside the cluster by assigning them their Continue reading