Most Google Kubernetes Engine (GKE) clusters work perfectly fine—until the moment they don’t.
As a Principal Architect, I’ve seen countless teams build “functional” workloads that pass every CI/CD test but crumble during a routine GKE control plane upgrade or a minor zonal disruption. There is a massive chasm between “running Kubernetes” and achieving Operational Excellence.
Running Kubernetes is about deployment. Operational Excellence is about predictable behavior under duress. It’s about ensuring that when a node fails, or the Cluster Autoscaler kicks in, your users never notice.
In a managed environment like GKE, Google handles the master nodes, but the “Operational Excellence” of the data plane—your workloads—is entirely on you.
I’ve audited environments where clusters were configured with the latest Autopilot features, yet a simple node auto-repair caused a cascading failure. Why? Because the workloads weren’t architected for the Voluntary Disruptions that are inherent to cloud-native life. Node upgrades, scaling events, and bin-packing are active, moving parts. If your workload isn’t designed to be “evicted” gracefully, it isn’t production-ready.
Resilience isn’t a single setting; it’s a system. I focus on four specific pillars that must interact perfectly to create a stable environment:
Pod Disruption Budgets (PDBs): Your availability guarantees.
Topology Spread Constraints: Your blast radius control.
Probes: Your application’s truth-telling mechanism.
Resource Quotas & Limits: Your predictability engine.
PDBs are your contract with the Kubernetes scheduler. They define how many replicas of a service must remain active during a voluntary disruption (like a GKE version upgrade).
The Upgrade Trap: Without a PDB, GKE might drain nodes too aggressively, taking your service below its minimum functional capacity.
Anti-Pattern Alert: Never set minAvailable: 100% or maxUnavailable: 0. This creates a “deadlock” where GKE can never upgrade your nodes because it can’t legally evict a single pod.
Workload Design: For stateless APIs, I typically recommend a maxUnavailable: 25%. For a single-replica service? A PDB is useless—you need to architect for at least two replicas to achieve resilience.
In GKE, your nodes are spread across zones. However, the default scheduler is often too “greedy,” packing pods onto the same node to save money. If that node dies, your entire service dies.
Topology Spread Constraints allow us to mandate a “skew.” By setting a maxSkew: 1 across topology.kubernetes.io/zone, we force Kubernetes to balance pods evenly across North American GCP zones.
Noisy Neighbors: By spreading across nodes, we mitigate the risk of one “heavy” pod starving others on the same hardware.
Autoscaling Interaction: Proper spread ensures that when the Cluster Autoscaler adds capacity, it does so in a way that maintains your high-availability (HA) posture.
Probes are how your application communicates its health to GKE. Most developers use a simple HTTP 200 check, but excellence requires more nuance.
Liveness: “Am I dead?” (If this fails, Kube kills the pod). Avoid putting database checks here, or a temporary DB flicker will cause a cluster-wide restart loop.
Readiness: “Am I ready for traffic?” This is for warming up caches or waiting for dependencies.
Startup Probes: These are the most underrated. For “heavy” apps (like JVM-based microservices), startup probes prevent the liveness probe from killing the pod before it even finishes booting.
Predictability is the prerequisite for stability. If you don’t define Requests, the scheduler is flying blind, leading to “bin-packing” nightmares.
Requests: This is what the scheduler uses to find a home for your pod.
Limits: This is the “ceiling.” Without limits, a memory leak in one pod can trigger an Out-Of-Memory (OOM) event that takes down the entire node.
The FinOps Angle: Accurate requests allow GKE’s Vertical Pod Autoscaler (VPA) to work effectively, ensuring you aren’t paying for “Ghost CPU” that you never actually use.
These four pillars are symbiotic.
PDBs without Spread Constraints mean you might have 10 pods, but if they are all on one node that GKE needs to upgrade, your PDB will be violated or your service will crash.
Good Probes with bad Quotas lead to pods that look “healthy” but are being throttled into uselessness by the CPU scheduler.
Scenario: A GKE Control Plane Upgrade at Peak Traffic.
Without Excellence: GKE drains a node; the pods move to another node that is already at 95% capacity; that node crashes; a cascading failure begins.
With Excellence: The PDB limits the rate of eviction. Topology Spread ensures pods move to a different zone. Resource Requests ensure the new node has guaranteed room. Readiness Probes ensure traffic only hits the new pods once they are fully initialized.
In a mature “Public Brain” model, we don’t expect every developer to be a Kubernetes expert. This is where Platform Engineering comes in.
We enforce these four pillars through Policy-as-Code (using OPA Gatekeeper or Kyverno) and Golden Paths. A developer should simply state “I have a web app,” and the platform should automatically inject opinionated PDBs and Spread Constraints.
If you are building on GKE today, start with these “Principal-level” defaults:
PDB: maxUnavailable: 1 for small sets; 25% for large ones.
Topology: Always use whenUnsatisfiable: DoNotSchedule for production workloads to ensure zonal spread.
Probes: Use a startupProbe with a generous failure threshold for any app taking $>10s$ to boot.
Resources: Set requests == limits for memory to prevent OOM kills during spikes.
Operational Excellence on GKE is not about 100% uptime—that’s an impossibility. It is about controlled failure. It’s about building a system that is boring, predictable, and fails safely.
Is your GKE environment architected for the chaos of a production environment?
Next Step: I can perform a GKE Resilience Audit of your current manifest files to identify where your “Paved Road” might have potholes. Would you like me to start by reviewing your current PDB and Resource Quota strategies?
