Spot VMs and Preemptible Capacity: Cut GKE Compute Costs 60–91% Without Downtime | 2026

What Are Spot VMs and Preemptible VMs?

Spot VMs are surplus Compute Engine capacity sold at deeply discounted prices. They are the modern replacement for preemptible VMs, offering the same low cost without the 24-hour maximum runtime limit. Google Cloud can reclaim Spot capacity with a 25-second shutdown warning, making them ideal for fault-tolerant, stateless, or interruptible workloads.

Think of Spot VMs like unsold airline seats sold at the gate for a steep discount. You get the same plane, same seat, and same service — but the airline can bump you at any moment if a full-fare passenger shows up. If you pack light and don't mind switching seats, you fly for a fraction of the price.

Why Use Spot VMs for GKE in Production and Non-Production?

On-demand Compute Engine nodes dominate Kubernetes infrastructure bills. In non-production environments, clusters often sit underutilized but still burn budget 24/7. Teams either overspend or under-provision, slowing down development and testing cycles.

Spot VMs let you scale node pools cheaply without sacrificing cluster availability. For non-prod, you can run entire clusters on Spot. For production, you can burst batch or stateless workloads onto Spot while keeping critical services on stable on-demand nodes. Combine Spot nodes with [Event-Driven VM Auto-Scheduling](/tutorial/event-driven-vm-auto-scheduling-cloud-functions-pubsub-scheduler) to turn off non-prod compute automatically on off-hours. To secure outbound connections from Spot workloads, see our guide on [Static Egress Gateways](/tutorial/cloud-run-gke-static-egress-ip-terraform-serverless-vpc-access-nat), and enforce zero-trust perimeter security with [VPC Service Controls](/tutorial/terraform-vpc-service-controls-custom-module-guide).

Prerequisites

• GCP project with billing enabled and GKE API activated
• gcloud CLI v450.0.0+ authenticated with `gcloud auth login`
• Terraform CLI v1.5.0+ — `terraform --version`
• IAM roles: `roles/container.admin` and `roles/compute.admin` on the project
• kubectl configured to access the target GKE cluster
• A stateless or fault-tolerant workload to deploy on Spot nodes

Step-by-Step Guide

Step 1: Enable Required GCP APIs

Activating Container, Compute, and Cloud Resource Manager APIs so Terraform can provision GKE clusters and node pools. GKE depends on the Compute Engine API for nodes and the Container API for cluster control plane operations.

resource "google_project_service" "required_apis" {
  for_each = toset([
    "container.googleapis.com",
    "compute.googleapis.com",
    "cloudresourcemanager.googleapis.com",
  ])

  project            = var.project_id
  service            = each.key
  disable_on_destroy = false
}

Step 2: Define Variables and Terraform Backend

Declaring typed variables for project, region, cluster name, and node pool configuration. Explicit variables make the module reusable across dev, staging, and demo environments.

variable "project_id" {
  description = "GCP Project ID"
  type        = string
}

variable "region" {
  description = "GKE cluster region"
  type        = string
  default     = "europe-west1"
}

variable "cluster_name" {
  description = "GKE cluster name"
  type        = string
  default     = "gke-nonprod-spot"
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
  backend "gcs" {
    bucket = "my-org-tfstate"
    prefix = "gke-spot/nonprod"
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

Step 3: Create the GKE Cluster and Stable System Node Pool

Provisioning a small on-demand node pool for kube-system, ingress controllers, and monitoring agents that must not be interrupted. Critical cluster infrastructure should not run on Spot capacity. A dedicated system pool keeps DNS, metrics, and ingress available during Spot preemption.

resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.region

  release_channel {
    channel = "REGULAR"
  }

  network    = "default"
  subnetwork = "default"

  # Remove the default node pool so we can manage pools explicitly
  remove_default_node_pool = true
  initial_node_count       = 1

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
}

resource "google_container_node_pool" "system" {
  name     = "system-pool"
  location = var.region
  cluster  = google_container_cluster.primary.name

  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  node_config {
    machine_type = "e2-medium"
    spot         = false

    labels = {
      node-type = "system"
    }

    taint {
      key    = "node-type"
      value  = "system"
      effect = "NO_SCHEDULE"
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

Step 4: Add a Spot VM Node Pool for Workloads

Creating a separate node pool where every node is a Spot VM. GKE automatically applies the `cloud.google.com/gke-spot` taint to these nodes. Isolating Spot nodes prevents interruptible workloads from co-locating with critical system pods and makes cost attribution simple.

resource "google_container_node_pool" "workload_spot" {
  name     = "workload-spot"
  location = var.region
  cluster  = google_container_cluster.primary.name

  autoscaling {
    min_node_count = 0
    max_node_count = 10
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  upgrade_settings {
    max_surge       = 2
    max_unavailable = 0
  }

  node_config {
    machine_type = "e2-standard-4"
    spot         = true

    labels = {
      node-type = "workload-spot"
    }

    # GKE automatically adds cloud.google.com/gke-spot=true taint,
    # but adding it explicitly documents intent and survives provider drift.
    taint {
      key    = "cloud.google.com/gke-spot"
      value  = "true"
      effect = "NO_SCHEDULE"
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

Step 5: Deploy Workloads with Spot Tolerations and Node Affinity

Configuring pods to tolerate the Spot taint and prefer scheduling onto Spot nodes. Without tolerations, the Kubernetes scheduler will never place pods on the Spot node pool because of the NO_SCHEDULE taint.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      tolerations:
        - key: "cloud.google.com/gke-spot"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values:
                      - workload-spot
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - api
                topologyKey: kubernetes.io/hostname
      containers:
        - name: api
          image: gcr.io/PROJECT_ID/api:v1.0.0
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 15"]
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"

Step 6: Protect Availability with a PodDisruptionBudget

Adding a PodDisruptionBudget that ensures at least two replicas remain available during voluntary disruptions such as Spot preemption or node upgrades. Without a PDB, Kubernetes can evict all replicas simultaneously during preemption, causing a brief outage even though capacity will return seconds later.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: default
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Step 7: Configure Graceful Shutdown and Monitor Preemptions

Setting `terminationGracePeriodSeconds` and capturing the preemption event logged by Compute Engine so the application can drain connections and exit cleanly. GKE sends a SIGTERM when a Spot node is preempted. The application has ~25 seconds to finish in-flight work before a SIGKILL.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          # Existing container spec ...
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 20"]

---
# Query recent preemption events in Cloud Logging
gcloud logging read 'protoPayload.serviceName="compute.googleapis.com"
  AND protoPayload.methodName="v1.compute.instances.preempted"'
  --project=${PROJECT_ID} --limit=10 --format=json

Verification & Health Check

Best Practices

• Run System Pods on a Non-Spot Pool
• Use Preferred Node Affinity, Not Required
• Set Realistic PodDisruptionBudgets
• Handle the Preemption Signal in Application Code

Common Mistakes

• {"errorCode":"PODS_UNABLE_TO_SCHEDULE","symptoms":"Workloads remain in Pending state after deploying to the Spot node pool.","rootCause":"The pod spec is missing the `cloud.google.com/gke-spot=true` toleration that matches the Spot node taint.","fixCommand":"kubectl edit deployment api and add the toleration for cloud.google.com/gke-spot","code":"tolerations:\n - key: \"cloud.google.com/gke-spot\"\n operator: \"Equal\"\n value: \"true\"\n effect: \"NoSchedule\"\n","language":"yaml","filename":"workload.yaml","prevention":"Include the Spot toleration in your base Helm chart or Kustomize overlay so every Spot workload inherits it automatically."}
• {"errorCode":"SERVICE_UNAVAILABLE_DURING_PREEMPTION","symptoms":"HTTP 503 errors spike when a Spot node is reclaimed, even with replicas > 1.","rootCause":"All replicas were scheduled on the same Spot node, or no PodDisruptionBudget was configured.","fixCommand":"kubectl apply -f pdb.yaml","code":"apiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: api-pdb\nspec:\n minAvailable: 2\n selector:\n matchLabels:\n app: api\n","language":"yaml","filename":"pdb.yaml","prevention":"Use podAntiAffinity on `topologyKey: kubernetes.io/hostname` and spread pods across zones with topologySpreadConstraints."}
• {"errorCode":"CONTAINER_KILLED_DURING_SHUTDOWN","symptoms":"Cloud Logging shows SIGKILL after 30 seconds; active requests are dropped.","rootCause":"The container did not finish draining within the default `terminationGracePeriodSeconds`.","fixCommand":"kubectl patch deployment api -p '{\"spec\":{\"template\":{\"spec\":{\"terminationGracePeriodSeconds\":60}}}}'","code":"spec:\n terminationGracePeriodSeconds: 60\n containers:\n - name: api\n lifecycle:\n preStop:\n exec:\n command: [\"/bin/sh\", \"-c\", \"sleep 25\"]\n","language":"yaml","filename":"workload.yaml","prevention":"Profile worst-case request duration and set terminationGracePeriodSeconds to at least 2x that value."}
• {"errorCode":"CLUSTER_AUTOSCALER_NOT_SCALING_SPOT","symptoms":"The Spot node pool stays at min_node_count even when pods are pending.","rootCause":"Cluster autoscaler may be disabled, or pending pods cannot tolerate the Spot taint.","fixCommand":"gcloud container clusters update ${CLUSTER_NAME} --region=${REGION} --enable-autoscaling","code":"resource \"google_container_cluster\" \"primary\" {\n # Cluster-level autoscaler is enabled by default for AUTOPILOT.\n # For STANDARD, add:\n cluster_autoscaling {\n enabled = true\n }\n}\n","language":"hcl","filename":"main.tf","prevention":"Verify autoscaler is enabled, node pool autoscaling min/max are set, and pod specs include the Spot toleration."}

Cost Analysis

References

• Spot VMs | Compute Engine Documentation
• Preemptible VMs | Compute Engine Documentation
• Create a node pool with Spot VMs | GKE Documentation
• PodDisruptionBudget | Kubernetes Documentation
• google_container_node_pool | Terraform Registry
• GKE Pricing | Google Cloud