GKE Autopilot Production Checklist: PDBs, HPA/VPA, Spot Pods & Cost Controls | 2026

What Is a GKE Autopilot Production Checklist?

GKE Autopilot removes node management — Google provisions, sizes, upgrades, and repairs nodes for you — but it does not remove workload-level responsibility. A production checklist covers the controls that still belong to you: PodDisruptionBudgets that survive voluntary disruptions, HPA/VPA autoscaling tuned to Autopilot's per-request billing, spot pod scheduling for batch-tolerant work, and cost guardrails that prevent oversized requests from silently inflating the bill.

Think of Autopilot like a fully serviced rental car — you never change the oil or rotate the tires, but you still choose the route (HPA), keep a spare seat policy (PDB), pick the cheaper fuel grade when the trip allows it (spot pods), and watch the fuel gauge (requests-based billing), because the rental company charges you for what you booked, not what you burned.

Why Does Autopilot Still Need a Production Checklist?

Teams assume Autopilot's managed nodes mean managed reliability and cost. In practice, the defaults hurt you twice: a PDB with `minAvailable: 1` on a single-replica deployment blocks node upgrades and scale-to-zero indefinitely, and unreviewed resource requests are billed in full 24/7 regardless of actual utilization — a deployment requesting 2 vCPU but using 200m costs 10x what it should, every hour of every day.

A short, opinionated checklist closes the gap: `maxUnavailable`-based PDBs that never block platform operations, HPA for traffic elasticity with VPA kept in recommendation mode to keep requests honest, spot pods for the tolerant share of capacity, and quotas plus budgets as financial guardrails. For the same discount story at the VM level, see [Spot VMs and Preemptible Capacity for GKE](/tutorial/spot-vms-preemptible-gke-cost-optimization), and for organization-wide guardrails see [GCP Organization Policies to Enable by Default](/tutorial/gcp-organization-policies-defaults-terraform-module).

Prerequisites

• GCP project with billing enabled and `container.googleapis.com` API enabled
• gcloud CLI v450.0+ and kubectl v1.28+ installed and authenticated
• `roles/container.admin` on the project (or GKE cluster admin equivalent)
• An existing Autopilot cluster, or permission to create one (regional, release channel REGULAR recommended)

Step-by-Step Guide

Step 1: Create or Audit the Autopilot Cluster Baseline

Provision a regional Autopilot cluster on the REGULAR release channel, or audit an existing one — release channel, maintenance windows, and cost-allocation usage metering are the cluster-level decisions everything else builds on. Release channel choice determines upgrade cadence (which exercises your PDBs), and enabling GKE cost allocation at cluster creation gives you namespace-level spend data you cannot retroactively recover.

gcloud container clusters create-auto prod-autopilot \
  --region=europe-west1 \
  --release-channel=regular \
  --maintenance-window=2026-08-23T02:00:00Z \
  --enable-cost-allocation \
  --project=$PROJECT_ID

gcloud container clusters get-credentials prod-autopilot \
  --region=europe-west1 --project=$PROJECT_ID

Step 2: Right-Size Resource Requests Before Anything Else

Set explicit CPU and memory requests on every container, near real p95 usage plus headroom. Autopilot rounds requests up to its minimums and bills every requested millicore, so this single manifest decision drives both reliability (scheduling) and cost. In Standard GKE, oversized requests waste bin-packing efficiency; in Autopilot they are literally the invoice. A request 2x too large doubles your pod bill forever.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/payments-api:1.4.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

Step 3: Configure PDBs That Never Block Upgrades

Add a PodDisruptionBudget per production workload using `maxUnavailable` instead of `minAvailable`, sized relative to replica count so voluntary disruptions (upgrades, scale-down, spot eviction) always have headroom. `minAvailable: N` with N equal to replica count makes disruptions impossible — GKE upgrades stall for hours, and scale-to-zero can never drain the last pod. `maxUnavailable: 1` scales with your deployment and never fully blocks the platform.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api-pdb
  namespace: payments
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: payments-api

Step 4: Set Up HPA on autoscaling/v2

Configure horizontal autoscaling on CPU utilization with scale-down stabilization, targeting the Deployment — Autopilot's metrics pipeline is built in, so no metrics-server install is needed. HPA is your traffic elasticity layer. On Autopilot it pairs naturally with per-request billing — you pay for extra replicas only while they exist, and stabilization windows prevent thrash-driven cost spikes.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api-hpa
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

Step 5: Add VPA in Recommendation-Only Mode

Deploy a VerticalPodAutoscaler with `updateMode: Off` so it continuously recommends right-sized requests without mutating pods — you review and apply recommendations through your normal CI/CD. Running HPA and VPA in `Auto` mode on the same CPU metric is a documented conflict (they fight over the same signal). `Off` mode gives you VPA's right-sizing intelligence — the single best cost lever on Autopilot — with zero surprise restarts.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payments-api-vpa
  namespace: payments
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  updatePolicy:
    updateMode: "Off"

Step 6: Shift Tolerant Replicas to Spot Pods

Add a spot-based replica set (or a second Deployment) using the `cloud.google.com/gke-spot=true` node selector with graceful shutdown, so 60–91% cheaper capacity absorbs baseline and overflow load while on-demand replicas guarantee the floor. Spot pods are the largest single discount lever in Autopilot. Used without a fallback, an eviction wave takes your service down; used with a minimum on-demand floor, evictions are a cost event, not an incident.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api-spot
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
      capacity: spot
  template:
    metadata:
      labels:
        app: payments-api
        capacity: spot
    spec:
      nodeSelector:
        cloud.google.com/gke-spot: "true"
      terminationGracePeriodSeconds: 25
      containers:
        - name: api
          image: europe-west1-docker.pkg.dev/PROJECT_ID/apps/payments-api:1.4.2
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]

Step 7: Add Namespace Quotas and Priority Classes

Constrain blast radius with a ResourceQuota per namespace (caps total requested CPU/memory, hence cost) and a PriorityClass so critical on-demand pods win scheduling over batch spot workloads. Autopilot bills what pods request; a ResourceQuota on requests is therefore a hard financial ceiling per namespace, not just a scheduling convenience.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: payments-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "16"
    requests.memory: 32Gi
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: production-critical
value: 1000000
globalDefault: false
description: "User-facing on-demand replicas; preempt batch and spot overflow"

Step 8: Wire Cost Visibility and Budget Alerts

Turn cluster cost allocation data into action — query namespace-level spend from the billing export and attach a budget alert scoped to the project, so an Autopilot cost anomaly pages the owners before the invoice arrives. Autopilot's per-request billing makes costs extremely predictable — which means any deviation is signal, not noise. A 2x week-over-week jump is almost always a request change or a runaway HPA max, and both are cheap to fix when caught early.

resource "google_billing_budget" "gke_autopilot_prod" {
  billing_account = var.billing_account_id
  display_name    = "gke-autopilot-prod-monthly"

  budget_filter {
    projects = ["projects/${data.google_project.prod.number}"]
    services = ["services/6F81-5844-456A"] # Kubernetes Engine
  }

  amount {
    specified_amount {
      currency_code = "EUR"
      units         = "2500"
    }
  }

  threshold_rules { threshold_percent = 0.5 }
  threshold_rules { threshold_percent = 0.8 }
  threshold_rules { threshold_percent = 1.0 }
}

Step 9: Run the Pre-Production Verification Suite

Before calling the workload production-ready, execute a five-minute verification pass: PDB status, HPA metrics, VPA recommendations, spot scheduling, and a live drain simulation. Checklists fail quietly — a PDB that exists but matches no pods, or an HPA pointing at the wrong Deployment name, looks green in `kubectl get` until the first incident. Explicit verification is the difference between documented and real resilience.

# PDB is healthy and matches pods
kubectl get pdb -n payments
kubectl describe pdb payments-api-pdb -n payments | grep -E "Allowed|Current"

# HPA sees metrics
kubectl get hpa -n payments

# VPA has recommendations (after 24-48h)
kubectl describe vpa payments-api-vpa -n payments | grep -A6 "Container Recommendations"

# Spot pods actually landed on spot capacity
kubectl get pods -n payments -l capacity=spot \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}'

# Simulate voluntary disruption — should succeed within PDB limits
kubectl drain $(kubectl get nodes -o name | head -1) \
  --ignore-daemonsets --delete-emptydir-data --dry-run=server

Verification & Health Check

Best Practices

• PDB with maxUnavailable, Not minAvailable
• HPA and VPA in Auto Mode on the Same Metric
• Spot Pods Without an On-Demand Floor
• Requests Set Once and Forgotten

Common Mistakes

• {"errorCode":"UPGRADE_STUCK / NODE_DRAIN_TIMEOUT","symptoms":"Cluster upgrade operations pending for days; nodes stuck in `Ready,SchedulingDisabled`.","rootCause":"A PDB with `minAvailable` equal to (or greater than) current replicas makes voluntary disruption impossible, so node drains can never complete.","fixCommand":"kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed\n","code":"apiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: payments-api-pdb\n namespace: payments\nspec:\n maxUnavailable: 1\n selector:\n matchLabels:\n app: payments-api\n","language":"yaml","filename":"fix-pdb.yaml","prevention":"Standardize on maxUnavailable in all PDB templates and alert on `disruptionsAllowed == 0` at steady state."}
• {"errorCode":"HPA_NO_METRICS","symptoms":"`kubectl get hpa` shows `<unknown>/70%` under TARGETS indefinitely.","rootCause":"The scaleTargetRef name or apiVersion does not match the Deployment, or the pods are crash-looping and emit no metrics.","fixCommand":"kubectl describe hpa payments-api-hpa -n payments | grep -A5 Events\n","code":"spec:\n scaleTargetRef:\n apiVersion: apps/v1 # not extensions/v1beta1\n kind: Deployment\n name: payments-api # must match metadata.name exactly\n","language":"yaml","filename":"fix-hpa.yaml","prevention":"Keep HPA and Deployment in the same Helm chart/kustomize overlay so they are applied and validated together."}
• {"errorCode":"SPOT_EVICTION_OUTAGE","symptoms":"Sudden latency spike or 5xx burst correlating with GKE node preemption events in Cloud Logging.","rootCause":"All replicas scheduled on spot capacity; a regional spot reclamation evicted the entire serving set at once.","fixCommand":"kubectl get pods -n payments -o wide | awk '{print $7}' | sort | uniq -c\n","code":"# Guaranteed on-demand floor behind the same Service\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: payments-api\nspec:\n replicas: 2 # on-demand, no gke-spot selector\n","language":"yaml","filename":"fix-spot-floor.yaml","prevention":"Enforce a minimum on-demand replica count in CI policy, and alert on 100%-spot serving state for user-facing Services."}
• {"errorCode":"AUTOPILOT_BILL_SPIKE","symptoms":"Kubernetes Engine line item jumps 2–3x week over week with no traffic change.","rootCause":"A merged change raised resource requests (billed 24/7 in Autopilot) or HPA maxReplicas let a retry storm scale to the ceiling and stay there.","fixCommand":"kubectl get vpa payments-api-vpa -n payments \\\n -o jsonpath='{.status.recommendation.containerRecommendations[0].target}'\n","code":"# Cap the blast radius at the HPA level\nspec:\n minReplicas: 2\n maxReplicas: 20 # review against quota and budget\n behavior:\n scaleDown:\n stabilizationWindowSeconds: 300\n","language":"yaml","filename":"fix-hpa-cap.yaml","prevention":"Require VPA-target comparison in PR review for any request change, and alert on HPA running at maxReplicas for more than 30 minutes."}

Cost Analysis

References

• GKE Autopilot overview and resource minimums
• Configure PodDisruptionBudgets in GKE
• Horizontal Pod autoscaling in GKE
• Vertical Pod autoscaling in GKE
• Deploy Spot Pods in GKE Autopilot
• GKE Autopilot pricing (per-pod requests)