Cost Guardrails

Audience: anyone who can spend money in our GCP account, or who can write code that spends money. Read after 00-overview.md.

Purpose: to protect us from catastrophic surprise bills. The GCP/CNTXT account terms have been described to us as “cut your head off” — if we overspend, the charge hits the card immediately, no forgiveness. This document defines the layered protections that have to be wired in before any application code or expensive resource ships.

These guardrails are M01 deliverables. Don’t deploy anything before they’re active.

1. The stakes in plain terms

A misconfigured Kubernetes autoscaler, a runaway BigQuery query, a forgotten GPU node, a busted Pub/Sub consumer creating egress, or an attacker triggering inference loops — any one of these can rack up four-figure bills in hours. With consumer-grade billing pricing it’s an annoyance; with our terms it’s an existential threat to the project’s runway.

The mitigations below are not optional and they are not aspirational. They are installed before the resource being protected exists.

2. Five layers of defense

We do not rely on any single control. The layers compound: if one fails, the others still hold.

3. Setup — L1: Budget and alerts

Wired in M01, before any other GCP resource.

3.1 Budget configuration

• Project budget: monthly cap at $4,000 USD for the pilot phase, with alerts as below. The cap is the alert threshold — GCP doesn’t hard-stop spending at the budget. The actual hard ceiling is the quota stack (L2). The cap reflects (a) me-central2 (Dammam) pricing being ~60% above us-central1 baseline, (b) the GPU pricing reality that only L4 is available in-region (~$1,000/mo per node 24/7), and (c) the operational policy that the cloud Tier 2 GPU stays always-warm 24/7/365 because fires happen year-round and 4-7 minute cold-starts on a fire-detection pipeline are unacceptable. Tier 0 + anomaly autoencoder run on the building’s edge device (CPU only) — removing the second cloud GPU.
• Per-service budgets for high-risk SKUs:
  • GPU compute: $1,500/month cap. L4 is the only GPU available in region; one always-warm Tier 2 L4 is ~$1,000/mo. Sub-budget leaves headroom for a second GPU within budget if testing shows we need more LLM throughput.
  • Egress: $200/month cap.
  • BigQuery query bytes: $100/month cap.
  • Cloud Storage: $200/month cap.

Budgets are set via Terraform (modules/billing-guardrails). The billing-alerts Pub/Sub topic receives every alert; it’s the audit-trail destination for postmortems even though there’s no programmatic consumer attached (see L3 below).

3.2 Alert thresholds

For each budget, alerts fire at:

• 25% (informational, email)
• 50% (email + Slack)
• 75% (email + Slack + SMS to Melissa)
• 90% (everyone, plus a Pub/Sub message published to billing-alerts for the audit trail)
• 100% (everyone — paged to phone)
• 110% (escalation: senior leadership, the bills have exceeded plan)
• 120% (emergency: kill non-critical workloads)

The exact recipients live in infra/terraform/billing/notifications.tf (to be written in M01).

3.3 Daily anomaly detection

Cloud Billing has an “anomalous spend” detector that flags day-over-day spikes. Enable it. Wire its notification into the same Pub/Sub topic.

4. Setup — L2: Quotas

Quotas are the hard fence. The GCP API refuses to provision a resource beyond a quota — no matter how badly the autoscaler wants to. This is the layer attackers and bugs cannot defeat.

4.1 Initial quota request profile

Request these specific quotas from CNTXT/Google when the account activates. Default GCP quotas are usually higher than we want; we lower them deliberately:

Quotas can be raised, but raising requires a written ticket. That friction is the point: a “raise quota” request is the cost-review moment.

4.2 Quotas as code

Quota state is captured in Terraform: infra/terraform/envs/{ops,pilot-prod}/quotas.tf. A planned Cloud Build job (or simple scheduled Cloud Run) diffs actual against the Terraform-declared values daily and alerts on drift. Anyone who raises a quota outside the Terraform path gets caught within a day. See Managing quotas for the current 11-cap setup on each project.

5. Setup — L3: Automated kill switches — deferred

Considered and deferred for the pilot. See decision log: 2026-05-14 — No auto-kill-switch service for the pilot.

The original plan called for a Cloud Run service that would subscribe to the billing-alert Pub/Sub topic and auto-execute predefined responses (scale Tier-2 LLM to 0, revoke BigQuery permissions, switch storage lifecycle, etc.) when budget thresholds were crossed.

Why we’re not building it for the pilot:

• Quotas already bound catastrophe. With NVIDIA_L4_GPUS = 2 on wiqaia, the absolute maximum possible GPU spend is 2 × $0.896/hr × 730 hr ≈ $1,308/mo. The GCP API physically refuses a third GPU. The “million-dollar bill from a runaway autoscaler” scenario requires someone first raising the quota — a deliberate human action with audit trail.
• Budget alerts at 25/50/75/90/100/110/120% of the soft budget already give seven escalating notifications. Critical thresholds page to phone.
• A false-positive auto-kill is dangerous. Scaling the Tier-2 LLM to 0 during a real fire crisis compromises the platform’s safety guarantee. The crisis-state-veto design that addresses this adds complexity that has to be right — and we’re carrying it forever.

The defense stack at pilot scale is L1 (budget alerts) + L2 (quotas) + L4 (sizing discipline) + L5 (engineering hygiene) + human response. L3 stays absent until the math changes:

• Multi-building scale where quota math no longer bounds catastrophe at ~$1,300/mo, OR
• Multiple humans on call where alert chains can slip through, OR
• An incident where the absence of L3 actually costs us money

If any of those trigger, revisit per the decision log.

The billing-alerts Pub/Sub topic stays in modules/billing-guardrails — every alert still publishes there with 7-day retention, so historical alerts are queryable for incident postmortems via gcloud pubsub topics. We just don’t have a programmatic consumer attached.

6. Setup — L4: SKU and sizing discipline

The default cost-aware choices, applied at every level of the stack. These are not optimization; they are the baseline.

6.1 Compute defaults

6.2 AI-specific cost rules

• Tier 2 LLM GPU pod is always-warm 24/7/365 (minReplicas: 1, maxReplicas: 2). Scale-to-zero is incompatible with a fire-detection guarantee — fires happen year-round and 4-7 minute cold-starts are unacceptable. This is the single largest line item in the pilot budget (~$1,000/mo) and is deliberate.
• There is no second cloud GPU. The original Tier-1 cloud GPU was eliminated on 2026-05-13 — Tier 0 motion + the anomaly autoencoder moved to the edge device (CPU-only). See docs/plan/01-architecture-summary.md §0.
• Per-camera AI rate is capped at 1 fps by default. Higher rates require a per-camera config flag and trigger a cost-review.
• Tier 2 escalations are rate-limited both at the edge (per-camera cap, default 4/min — enforced before the frame leaves the building) and at the cloud service (per-building cap, default 60/min) to prevent a misbehaving edge from flooding the LLM.
• Streaming bandwidth is on-demand: cameras stream only when an operator is actively viewing or an alert is active. Idle = $0 bandwidth.
• Frame storage is not always-on. Only frames associated with escalations, alarms, or operator review are stored. Random frames don’t accumulate.

6.3 Network defaults

• All inter-service traffic on the GKE internal network (no public IPs except the API gateway).
• One Cloud Load Balancer per environment, not per service.
• Egress to known destinations (the edge boxes, the operator browsers via CDN) only. Block egress to anywhere else by default.
• Connection pooling everywhere (Postgres, Redis, Pub/Sub). No “new connection per request”.

7. Pilot budget — the explicit cost envelope

Region pricing reality — verified 2026-05-13. me-central2 (Dammam) is the most expensive GCP region globally (rank #42 of 42 by average hourly compute price). The average premium across compute SKUs is roughly +88% vs the cheapest GCP region (per CloudPrice’s regional comparison, fetched 2026-05-13). Earlier planning text in this document used “+60% over us-central1” as a working estimate — that holds for individual N2/E2 instances and pd-balanced disks, but the broader average across all SKU classes is closer to +88%. Use +88% as the planning baseline going forward, not +60%. Spot-checked single-resource examples: e2-small is $19.57/mo in me-central2 vs ~$12.23/mo in us-central1 (+60%); pd-balanced disk is $0.16/GB-month vs ~$0.10/GB-month elsewhere (+60%); attached static external IPs are $3.65/mo (GCP-wide rate; me-central2 doesn’t add a regional premium on top). GPU availability is also constrained: only NVIDIA L4 is available in me-central2, and only in zones -a and -c (NOT -b). T4 / A100 / H100 are not available in-region. CNTXT-specific markup on top of GCP list pricing has not been independently confirmed; assume list-price passthrough but verify with CNTXT before final acceptance.

Pilot estimates below reflect the me-central2 reality. Numbers are USD per month. Refined as soon as the first month of actual CNTXT billing data lands.

Architecture decision: edge-side anomaly detection. Original three-tier plan (edge motion / cloud Tier 1 GPU / cloud Tier 2 GPU) had two cloud GPUs always-warm at $2,000/mo. New two-tier plan runs Tier 0 motion filter + anomaly autoencoder on the building’s edge device (CPU-only inference, well within a modern desktop’s headroom) and uses a single cloud L4 for the Tier 2 multimodal LLM. Cuts cloud GPU bill by 50%, reduces bandwidth (fewer frames uploaded), gives slight resilience to cloud outages (local anomaly scoring continues; alert distribution still requires cloud).

On the always-warm cloud GPU choice: scale-to-zero means 4-7 minute cold-start latency before the LLM returns a verdict. Incompatible with the platform’s value proposition. The $1,000/mo always-warm cost is a deliberate decision to make sub-10-second verdicts the guarantee, not the happy path.

This cost is largely fixed regardless of camera count. One Tier-2 L4 sustains ~5-7 LLM inferences/sec with vLLM batching, which comfortably handles 8-10+ buildings’ worth of escalation load (40 cameras × 30 s baseline cadence per building ≈ 1.3 escalations/sec/building). So as we onboard buildings 2, 3, 4 — the $/camera/month for cloud GPU drops sharply:

Edge-device cost (motion + autoencoder) is paid by the customer (either bring-your-own to spec or buy ours), so it doesn’t show up on our cloud bill.

Budget cap (project-wide) — $4,000/month, with alerts at the usual thresholds (25/50/75/90/100/110/120%). Sized for the always-warm cloud GPU baseline ($1,000/mo, one L4 for the Tier 2 LLM) + the rest of the data plane (~$1,200/mo) + Hajj surge + buffer + headroom to briefly spin up a second L4 if testing shows we need more LLM throughput. GPU sub-budget — $1,500/month — leaves room for a brief second always-warm GPU node within budget if real telemetry justifies it.

The on-prem-GPU contingency is still relevant but less urgent. At $1,000/mo of always-warm cloud GPU × 12 months = $12K/year. One L4 workstation at ~$6K capex pays back in ~6 months. Worth evaluating once we onboard the second or third building, especially if Tier-2 throughput holds at the single-L4 baseline. See docs/plan/05-contingencies.md §1.

These numbers are the floor; we tighten them as real metrics arrive. Every milestone with infrastructure work updates this table in the same PR.

8. Cost-aware engineering hygiene

The above are mechanical controls. The cultural controls matter too.

8.1 Cost estimate accompanies every infrastructure change

Every Terraform PR and every Helm chart change carries a “Cost impact” section in the PR description:

## Cost impact
- New: Cloud SQL read replica (1 instance, db-n1-standard-2): ~$150/month
- Removed: nothing
- Net: +$150/month
- Notes: replica enables read-heavy analytics queries; offsets BigQuery scans

Reviewers check this. If a change has no cost section, it’s not approved.

8.2 The monthly cost review

First business day of each month: someone (Melissa or me) walks the Cloud Billing report and reconciles against the pilot budget table above. Items that drifted up explain themselves; unexplained drift is a ticket.

8.3 No idle GPUs, ever

GPU instances are the single biggest cost-explosion risk. Rules:

• No GPU node pool has minNodes > 0 outside the pilot window unless explicitly approved.
• Every GPU pod has a terminationGracePeriodSeconds ≤ 30 so it scales down fast.
• Tier-2 deployment uses HPA + Cluster Autoscaler to scale node-and-pod together on Pub/Sub queue depth.
• Nightly job confirms no GPU node has been up > 12 hours without serving requests in the last hour. Alert if it has.

8.4 The “is this expensive?” checklist

Before pushing any code that adds infrastructure or increases load, ask:

• Does this run continuously, or scale-to-zero?
• Is it on the cheapest viable SKU for its workload?
• Is its scaling bounded by quota?
• Does it write logs at a rate that will cost real money?
• Does it generate egress to public destinations?
• Does it write to storage that has no lifecycle policy?

A “no” on any of these is a review conversation, not a blocker — but it has to be conscious.

9. Cost dashboards

A single Cloud Monitoring dashboard at the canonical URL monitoring/dashboards/cost-overview:

• Daily spend, last 30 days, by service
• Daily spend, last 30 days, by label (environment, tenant)
• GPU node-hours used today
• BigQuery bytes scanned today
• Egress bytes today
• Storage GB by class
• Top 10 most-expensive resources right now

This dashboard is checked by whoever’s on rotation, at least daily during the pilot window.

10. Process: emergency cost response

If the 120% total-project alert fires:

1. Melissa and the cloud engineer hire paged. (Even at 3 AM.) The phone notification is the trigger.
2. Within 15 minutes, the on-call:
   • Opens the cost-overview dashboard.
   • Identifies the runaway resource (which budget popped, which SKU is climbing).
   • Disables it manually — typical interventions:
     • GPU runaway: kubectl scale deployment tier2-llm --replicas=0 (with the understanding that this dark-window kills cloud LLM analysis; the edge-side autoencoder continues running on the customer’s hardware regardless).
     • Egress runaway: add a deny rule to the egress firewall for the suspect destination.
     • BigQuery runaway: revoke the runaway service account’s bigquery.jobs.create permission.
     • Storage runaway: identify the bucket and switch its lifecycle to coldline immediately.
   • Communicates ETA on customer-visible features.
3. Post-incident: root cause analysis the same week, with a documented gap-closure in this file. If the same scenario could recur and the quotas + alerts + human response stack didn’t catch it fast enough, revisit the decision to skip L3 — see decisions.

11. What to do if you’re about to spin up something expensive

When in doubt:

• Estimate the cost before pressing apply.
• Apply in a temporary GCP project first (it has its own budget cap) and watch real billing data accumulate for an hour.
• Only then promote the change to the pilot project.
• Document the estimate vs actual in your PR description.

The fastest way to break this project is to be lazy about cost. Every other guardrail in this doc is downstream of that single insight.