Stateful Workloads on GKE: StatefulSets, Regional PDs & Backup for GKE | 2026

StatefulSets vs Deployments — Choosing the Right Workload Controller

A StatefulSet is a Kubernetes workload controller designed for applications requiring stable network identity, ordered deployment/scaling, and persistent storage. Unlike Deployments which treat pods as interchangeable, StatefulSets assign each pod a stable sticky hostname (e.g. `redis-0`, `redis-1`, `redis-2`) and a dedicated PersistentVolumeClaim.

Think of a Deployment like hotel rooms — guests are interchangeable and any empty room works for any guest. A StatefulSet is like apartment buildings with numbered units — resident `redis-0` always lives in apartment #0, keeps their own keys, and when they move out and return, they return to the same unit with the same furniture (PersistentVolume) still in place.

Regional PD vs Zonal PD vs Cloud Storage for GKE Persistent Volumes

Zonal Persistent Disks are pinned to a single zone. If that zone fails, pods cannot reschedule because the PV is inaccessible from other zones. This is a hard availability ceiling for stateful GKE workloads.

Regional Persistent Disks replicate data synchronously across two zones. When a zone fails, GKE automatically reschedules pods to the surviving zone, and the regional disk reattaches with no data loss. Compare with managed database alternatives like [Cloud SQL Postgres Private IP](/tutorial/cloud-sql-postgres-production-private-ip-ha-backups-terraform) and secure cluster traffic using [Cloud Service Mesh](/tutorial/cloud-service-mesh-gke-mtls-traffic-splitting-authorization-policies).

Prerequisites

• GKE cluster v1.28+ (Standard or Autopilot)
• Backup for GKE addon enabled on the cluster
• Terraform CLI v1.6+ and kubectl configured
• GCP Project with Compute Engine and Backup for GKE APIs enabled

Step-by-Step Guide

Step 1: Configure Regional SSD StorageClass in Terraform

Create a Kubernetes StorageClass provisioning Regional Persistent Disk SSDs across two availability zones. StorageClass parameters define the disk type and replication zones. Without `replication-type: regional-pd`, GKE provisions zonal disks by default.

# storageclass.tf — Regional SSD Persistent Disk StorageClass
resource "kubernetes_storage_class" "regional_ssd" {
  metadata {
    name = "regional-ssd"
  }

  storage_provisioner    = "pd.csi.storage.gke.io"
  reclaim_policy         = "Retain"
  allow_volume_expansion = true
  volume_binding_mode    = "WaitForFirstConsumer"

  parameters = {
    type              = "pd-ssd"
    "replication-type" = "regional-pd"
    zones             = "europe-west1-b,europe-west1-c"
  }
}

Step 2: Deploy StatefulSet with Headless Service and VolumeClaimTemplate

Create a Redis StatefulSet with 3 replicas using the Regional SSD StorageClass via VolumeClaimTemplates. StatefulSets combine stable pod identity and dedicated persistent volumes — each `redis-N` pod always reconnects to its own `data-redis-N` PVC containing its data.

# redis_statefulset.yaml — 3-replica Redis StatefulSet with Regional PD
apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  namespace: production
spec:
  clusterIP: None # Headless — creates per-pod DNS entries
  selector:
    app: redis
  ports:
    - port: 6379
      name: redis
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: production
spec:
  serviceName: redis-headless
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.2-alpine
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "500m"
              memory: "1Gi"
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: regional-ssd
        resources:
          requests:
            storage: 50Gi

Step 3: Verify Zone Resilience with Pod Disruption

Simulate a zone failure by cordoning all nodes in one zone and verify pod rescheduling to the surviving zone with regional PD reattachment. Testing zone failover in pre-production validates that Regional PDs actually reattach successfully and no data loss occurs during zone impairments.

# Step 1: Identify nodes in zone europe-west1-b
kubectl get nodes --label-columns=topology.kubernetes.io/zone

# Step 2: Cordon all nodes in the target zone (simulates zone failure)
kubectl cordon NODE_NAME_B_1 NODE_NAME_B_2

# Step 3: Delete redis-0 pod (which ran in zone B)
kubectl delete pod redis-0 -n production

# Step 4: Observe StatefulSet reschedule redis-0 to zone europe-west1-c
kubectl get pods -n production -w

# Expected output:
# redis-0   0/1   Pending   0   10s   (Waiting for Regional PD reattach)
# redis-0   1/1   Running   0   52s   (Rescheduled to zone C with data intact)

# Step 5: Uncordon nodes after test
kubectl uncordon NODE_NAME_B_1 NODE_NAME_B_2

Step 4: Enable Backup for GKE and Schedule Namespace Backups

Enable the Backup for GKE addon on the cluster and create a BackupPlan with daily schedule and 30-day retention. Regional PDs protect against zone failure but not against accidental data deletion or application corruption. Backup for GKE provides point-in-time recovery.

# backup_for_gke.tf — Enable Backup Addon and BackupPlan
resource "google_container_cluster" "prod" {
  # ... other cluster config
  addons_config {
    gke_backup_agent_config {
      enabled = true
    }
  }
}

resource "google_gke_backup_backup_plan" "production_backup" {
  name     = "production-daily-backup"
  cluster  = google_container_cluster.prod.id
  location = "europe-west1"
  project  = var.project_id

  backup_config {
    include_volume_data = true
    include_secrets     = true
    selected_namespaces {
      namespaces = ["production", "databases"]
    }
  }

  backup_schedule {
    cron_schedule = "0 2 * * *" # Daily at 02:00 UTC
  }

  retention_policy {
    backup_delete_lock_days = 7   # Cannot delete backups within 7 days
    backup_retain_days      = 30  # Keep 30 days of backups
  }
}

Step 5: Restore Namespace from Backup

Execute a namespace restore from a Backup for GKE snapshot into a new or existing namespace. Testing restore procedures verifies that backups are valid and your team can execute recovery under time pressure during production incidents.

# Step 1: List available backups
gcloud beta container backup-restore backups list \
  --backup-plan=production-daily-backup \
  --location=europe-west1 \
  --project=PROJECT_ID

# Step 2: Initiate namespace restore from a specific backup
gcloud beta container backup-restore restores create production-restore-20260829 \
  --backup=projects/PROJECT_ID/locations/europe-west1/backupPlans/production-daily-backup/backups/BACKUP_NAME \
  --restore-plan=production-restore-plan \
  --location=europe-west1 \
  --project=PROJECT_ID

# Step 3: Monitor restore progress
gcloud beta container backup-restore restores describe production-restore-20260829 \
  --restore-plan=production-restore-plan \
  --location=europe-west1

Verification & Health Check

Best Practices

• Set Reclaim Policy to Retain on Production StorageClasses
• Define PodDisruptionBudget Alongside Every StatefulSet

Common Mistakes

• {"errorCode":"REGIONAL_PD_ZONE_MISMATCH","symptoms":"PVC stuck in Pending with FailedAttachVolume: Regional PD zones do not match node zones.","rootCause":"StorageClass zones parameter specifies zones that don't match any available node zones.","fixCommand":"kubectl describe pvc data-redis-0 -n production","code":"# Fix StorageClass to match actual cluster node zones\nparameters = {\n type = \"pd-ssd\"\n \"replication-type\" = \"regional-pd\"\n zones = \"europe-west1-b,europe-west1-c\" # Must match cluster zones!\n}\n","language":"hcl","filename":"fix_storageclass.tf","prevention":"Verify cluster node zones with `kubectl get nodes -L topology.kubernetes.io/zone` before creating StorageClass."}

Cost Analysis

References

• StatefulSets in Kubernetes
• Regional Persistent Disks for GKE
• Backup for GKE Overview