Managing GCP quotas

Quotas are the first line of defense against a runaway-cost incident. They sit at the GCP API layer: when any call (Terraform, gcloud, Console, an SDK from a misbehaving app) would push past the cap, the API rejects it immediately. No spend ever happens. Budget alerts are the second line of defense; they fire after the bill is already accumulating.

We run two GCP projects, both with quota guardrails:

Both stay well below their $50 / $4,000 budget alert thresholds, so the budget alert is genuinely a backstop, not a trap that fires during normal operation.

What’s set today

Each project carries the same 11 quota preferences (10 compute + 1 logging), sized differently per project.

Where it lives in code

The pilot env’s TF is deliberately un-applied until cash is in the bank and the senior-engineer audit (CE-03) has run — see Cost tracker for the spend-zero state. Quotas were applied separately via gcloud so the guardrails are live now without any other resources being provisioned.

Adding or changing a quota

The clean path — via Terraform (preferred when state is initialized)

1. Edit the relevant quotas.tf. Add a new google_cloud_quotas_quota_preference block or change a preferred_value.
2. terraform plan from infra/terraform/envs/<env>/ to confirm the diff.
3. terraform apply.

If you’re decreasing a value by more than 30%, keep the ignore_safety_checks = "QUOTA_DECREASE_PERCENTAGE_TOO_HIGH" line. Otherwise the API rejects the change with a warning.

The pragmatic path — via gcloud (when TF state isn’t ready for the project)

This is how the pilot project works today.

gcloud beta quotas preferences create 
  --preference-id=pilot-l4-gpus-per-project-region-mec2 
  --project=wiqaia 
  --service=compute.googleapis.com 
  --quota-id=NVIDIA-L4-GPUS-per-project-region 
  --preferred-value=2 
  --dimensions=region=me-central2 
  --justification="WiqAIa+ Saudi Airlines pilot..." 
  --allow-high-percentage-quota-decrease

When the pilot env’s TF eventually applies, the existing preferences will be imported into Terraform state with terraform import (same as we did for wiqaia-ops originally). The names in pilot-prod/quotas.tf already match the gcloud-created preferences, so import is clean.

Required: enable cloudquotas.googleapis.com

The first quota preference you set on a project fails until you enable the Cloud Quotas API:

gcloud services enable cloudquotas.googleapis.com --project=<PROJECT>

For wiqaia-ops and wiqaia this is already done. For any future project, it’s a one-shot.

Required: provider config for Terraform

The Cloud Quotas API resource type only exists in the google-beta provider, not google. The provider config must look like:

# versions.tf
terraform {
  required_providers {
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 6.0"
    }
  }
}

provider "google-beta" {
  project               = var.project_id
  region                = var.region
  user_project_override = true     # ← essential
  billing_project       = var.project_id
}

The user_project_override = true + billing_project are load-bearing. Without them, Terraform makes Cloud Quotas API calls billed against gcloud’s OAuth client app project (Google’s internal 764086051850), which doesn’t have cloudquotas.googleapis.com enabled — so every call 403s with SERVICE_DISABLED. This was a real debugging adventure; documented here so future-us doesn’t repeat it.

And each resource needs an explicit provider = google-beta:

resource "google_cloud_quotas_quota_preference" "l4_gpus" {
  provider = google-beta
  # ...
}

Known quirk: preferred_value = 0 shows persistent drift

The Cloud Quotas API GET endpoint omits preferred_value from the response when it equals 0. So Terraform reads the state as “no preferred value set”, but the config says preferred_value = 0. Result: terraform plan shows a perpetual three-row in-place diff for the value-0 preferences (l4_gpus on ops, preemptible_cpus, preemptible_l4_gpus).

This is harmless. The server holds the right value; terraform apply re-asserts it idempotently; nothing changes. Documented in the comment block at the top of each quotas.tf so future contributors don’t chase it as a bug.

Workaround if it ever gets noisy: add lifecycle { ignore_changes = [quota_config[0].preferred_value] } to the three affected resources. We’ve deliberately not done this because then a real drift wouldn’t surface either.

Verifying what’s actually live

# Both projects (the names are scoped per project)
gcloud beta quotas preferences list --project=wiqaia-ops 
  --format="table(name.basename(),quotaId,quotaConfig.grantedValue,quotaConfig.preferredValue)"

gcloud beta quotas preferences list --project=wiqaia 
  --format="table(name.basename(),quotaId,quotaConfig.grantedValue,quotaConfig.preferredValue)"

grantedValue is what the API is enforcing. preferredValue is what we asked for. They should match (except for the value-0 quirk above where preferredValue is blank).

When to raise a cap

The GPU cap is the only one we treat with kid-glove care. Going from 0 → 2 or 2 → 3 should be near-instant (within Google’s defaults), but operationally:

• GPU cap: raise only with explicit decision-maker sign-off. The corresponding gke_cluster Terraform module also needs to be uncommented + populated — the cap and the module are both gates. Today (2026-05-14) the cap is 2 on the pilot, no GPU is actually provisioned.

For every other cap: raise when an apply hits the wall. Edit the preferred_value, apply, done. Within-default raises are processed by Google in seconds-to-minutes. Above-default raises require a manual review from GCP support (days, sometimes a week).

Related runbooks

• Cost tracker — current spend, where the budget alerts sit.
• Decisions — why the automated kill-switch was deferred at pilot scale.
• Checking GCP prices — how to verify the actual unit prices we used for the “worst case at cap” numbers above.