Decision log

Chronological record of meaningful architectural / scope decisions and why we made them. Append-only — earlier entries don’t get edited even when superseded; a new entry supersedes the old one with explicit reference. The point is to keep the reasoning visible to anyone joining the project later.

Format:

## YYYY-MM-DD — One-line title

**Status:** accepted | superseded by [link] | reversed
**Context:** what was the situation that forced the decision
**Considered:** the options we weighed
**Decided:** what we chose, in one sentence
**Why:** the reasoning, including the things that *would* change the answer
**Implications:** what this means for downstream work / what we're now NOT doing

2026-05-14 — Service mesh: Linkerd (not Istio)

Status: accepted

Context: The plan calls for mTLS between every internal service in the cluster (docs/plan/03-data-protection §3). The two viable options for a service mesh on GKE are Linkerd and Istio (the latter available as a Google-managed offering: Anthos Service Mesh).

Considered:

  1. Istio via Anthos Service Mesh (managed by Google). Feature-rich, deep observability, complex configuration model (VirtualService / DestinationRule / Gateway). ~$0.08/CPU-hour managed-control-plane fee on top of the existing GKE cluster cost.
  2. Istio self-managed. Same feature set, no managed fee, but we’re on the hook for upgrades.
  3. Linkerd. Smaller, simpler, lighter memory footprint per pod. Doesn’t have a managed offering on GKE — we run it ourselves. No managed fee.

Decided: Linkerd, self-managed.

Why:

  • We don’t need Istio’s traffic-shifting machinery. No A/B routing, no canary VirtualServices, no gateway federation. Just mTLS between services + zero-trust posture + observability.
  • Linkerd’s per-pod sidecar uses ~10-30 MB vs Istio’s ~100-200 MB. At pilot scale on e2-medium general nodes (4 GiB RAM total per node), this matters.
  • Operational simplicity. Linkerd’s reputation is “just works”; Istio’s reputation is “powerful but complex to operate.” For a solo lead + one cloud-engineer-hire team, simpler wins.
  • mTLS by default. Linkerd installs with automatic mTLS for every meshed pod — no policy configuration needed to meet our security baseline.
  • Cost. Saves the ASM managed-fee at our scale (small for the pilot, but pure overhead given we don’t use the features that justify it).

What would change the answer:

  • Multi-cluster federation. If we ever run clusters in multiple regions and need cross-cluster service discovery / traffic routing, Istio’s ecosystem is stronger. We’re single-cluster for the pilot and the foreseeable post-pilot expansion.
  • Heavy traffic-shifting needs. If we end up doing complex canary deployments or chaos-engineering experiments at the network layer, Istio’s tooling is richer.
  • Compliance audit that demands a managed mesh. Some NCA reviews prefer managed-by-Google for cloud-native components. Confirm during the audit.

Implications:

  • The gke-cluster Terraform module does NOT enable Anthos Service Mesh.
  • Linkerd installs via its CLI / Helm chart, applied to the cluster after first apply. Not part of Terraform (Linkerd’s install is post-cluster-bootstrap; managing it in Terraform’s kubernetes provider is brittle).
  • docs/plan/M01-foundations.md §7 updated to reflect the choice.
  • Operators reach the cluster API server through Linkerd’s mesh-gateway (mTLS-terminated at the perimeter). The external HTTPS LB terminates client TLS; Linkerd handles intra-cluster.

2026-05-14 — No auto-kill-switch service for the pilot

Status: accepted

Context: The original 02-cost-guardrails plan called for a Cloud Run service (the “L3 kill-switch”) that would subscribe to billing-alert Pub/Sub messages and auto-execute predefined responses — most importantly, scale the Tier-2 LLM deployment to 0 replicas at 100% of the GPU sub-budget. Mid-build we asked whether the kill-switch could itself be more dangerous than the bills it’s protecting against, given that scaling the LLM to zero during an active fire crisis would compromise the platform’s safety guarantee.

Considered:

  1. Build it as originally planned. Auto-kill at 100% of soft budget.
  2. Build a smarter version with crisis-state veto (don’t fire if an active alert exists in the past N hours), hard absolute thresholds rather than percentage-of-soft-budget, and per-category response gradations.
  3. Don’t build it. Rely on quotas + budget alerts + human response as the defense stack.

Decided: Don’t build it for the pilot.

Why: The math doesn’t justify the complexity at our scale.

  • Quotas are the actual hard ceiling. With NVIDIA_L4_GPUS = 2 the absolute maximum possible GPU spend is 2 × $0.896/hr × 730 hr = ~$1,308/mo. The API physically refuses to provision a third GPU. The “million-dollar bill” scenario requires someone first raising the quota — a deliberate human action with an audit trail.
  • Budget alerts at 25 / 50 / 75 / 90 / 100 / 110 / 120% of the soft budget already give humans seven escalating notifications. Critical thresholds page to Melissa’s phone.
  • Daily anomaly detection (built into Cloud Billing, free) flags day-over-day spikes.
  • The scenarios a kill-switch saves us from require simultaneous failure of (a) quotas being raised inappropriately, (b) human alert response, (c) Google’s anomaly detection. Plausible but rare.
  • The cost of a false-positive kill is high — scaling the Tier-2 LLM to 0 during a real fire compromises the platform’s safety guarantee. The crisis-state-veto design that addresses this adds complexity that has to be right.
  • The ongoing maintenance cost of a Cloud Run service, IAM, OIDC verification, action handlers, and tests is meaningful — and we’d be carrying it forever for a contingency that the math says is bounded by the quotas.

What would change the answer:

  • Multi-building pilot with larger budgets where the quota math no longer bounds catastrophe at ~$1,300/mo.
  • Multiple humans in the on-call rotation where it’s easier for an alert chain to slip through.
  • An incident where the absence of L3 cost us real money.

If any of those trigger, revisit. Until then the defense stack is quotas + alerts + human response.

Implications:

  • The infra/kill-switches/ scaffolding (Cloud Run service handler) is deleted from the repo.
  • 02-cost-guardrails.md §5 (the “L3 — Automated kill switches” section) is rewritten as “considered and deferred — see decisions log.”
  • Gitea issues #17 Kill-switch Cloud Run service and #30 CE-05: Kill-switch end-to-end verification are closed as won’t-do, with cross-links to this decision.
  • Contributor-brief CE-05 and CE-13 are dropped from the cloud-engineer scope.
  • The billing-alerts Pub/Sub topic stays in modules/billing-guardrails — it’s still the destination for budget-alert messages, it just doesn’t have a programmatic consumer for now. Pub/Sub message retention (7 days) means historical alerts are queryable for incident postmortems.
  • M01 verification item #5 (synthetic budget alert triggers kill-switch) is replaced by a simpler verification: synthetic budget alert publishes to the topic and is observable via gcloud pubsub topics.
WiqAIa+ Docs · Rizoma