Contingencies

Audience: the whole team. Read after 00-overview.md. Re-read whenever something feels stuck.

Purpose: to enumerate the most likely “things go wrong” scenarios and the pre-planned responses. The architecture is designed so that each of these is a deployment-mode switch or a driver swap, not a panic rewrite.

Every contingency below has three parts: the trigger (how we know we need to act), the response (what we do), and the cost (what it costs us in time, money, scope).

1. Dammam GPU quota / availability shortage (or just: Dammam GPU pricing makes on-prem more attractive)

Background — me-central2 GPU reality (verified 2026-05-13). Only NVIDIA L4 is available in me-central2, and only in zones -a and -c. T4 / A100 / H100 are not available. On-demand pricing for g2-standard-8 (single L4) is ~$1.37/hr ≈ $1,000/mo running 24/7. me-central2 is the most expensive GCP region globally, ranked #42 of 42, with a ~60% premium over us-central1 on common compute SKUs. This contingency is also worth activating purely on cost grounds even if quota lands cleanly — see “Cost-driven activation” below.

Trigger. Any of:

• CNTXT/Google cannot provision L4 capacity in me-central2 by the time M06 begins, or quota requests are denied or take too long. Specific signals: a quota-raise ticket open more than 5 business days; a compute.googleapis.com.GpuFamily denied at create time; CNTXT account manager indicating capacity constraints.
• The first month of real billing shows cloud GPU costs trending above $1,000/mo and we expect that to continue beyond the pilot — at which point on-prem capex pays back in 6-12 months at the new single-GPU baseline. Cost-driven activation.

Response. Deploy the Tier-2 LLM worker on a small on-premise GPU box installed at the customer site (the same edge device, if it has a GPU; otherwise a second box on the same LAN). The worker has been designed from M01 to support this as a first-class deployment mode, selected by configuration. Same code; different topology. The edge-side Tier 0 + anomaly autoencoder already runs on-site in both modes (CPU only), so this contingency only affects the cloud LLM hop.

The on-prem mode looks like:

                  Cloud (Dammam)
                  ──────────────
                  API · Postgres · Redis · Pub/Sub · Dashboard · Fleet Mgr
                  ▲
                  │ mTLS tunnel (same as edge supervisor)
                  ▼
                  On-prem AI box at the customer site
                  ───────────────────────────────────
                  Tier 2 LLM inference (L4 or 4070+) · MediaMTX
                  ▲
                  │ Loopback / LAN
                  ▼
                  Edge supervisor + edge-app containers (Tier 0 + autoencoder, CPU)
                  ▲
                  │ LAN (camera-side)
                  ▼
                  Cameras · Fire panel

Hardware options (any one):

For the Saudi Airlines pilot specifically: a single L4-equipped workstation is the right answer if we go this route. ~$6K capex, falls within the contract margin.

Code changes required. None — the contingency is selected by config. The Tier-2 worker reads WIQAIA_DEPLOYMENT_MODE=cloud|on_prem|hybrid and adjusts its storage backend (GCS → MinIO), its model loading path (Artifact Registry → local Docker daemon), and its tunneling (none → mTLS to cloud). All driver swaps via the abstractions in 04-abstractions-and-contracts.md.

Operational changes. The on-prem box runs the same edge supervisor (M09) that manages the regular edge containers — the Tier-2 LLM worker just becomes another container under its watch. No new fleet-management surface.

Cost. Capex ~$5–7K for the box; ongoing electricity and rack-space are the customer’s responsibility per the LOA template (provider-supplied facilities). Cloud GPU spend drops to $0 while in this mode. Math: at me-central2 list rates (~$1,000/mo for one always-warm L4), a $6K workstation pays back in 6 months. With only one cloud GPU in the 2-tier architecture (down from two in the original plan), the payback is the same per-GPU but the absolute monthly savings is half — still worth doing on cost grounds if we expect to run for 12+ months.

Time to switch. Order box → on-site → install → register → ~3 business days assuming hardware is in stock. We pre-order one workstation in M01 as insurance, with the understanding that if Dammam GPU quota lands, we sell or repurpose the unit.

2. Gemma 4 is not multimodal / not vision-capable / quality fails our test

Trigger. Gemma 4 ships text-only initially (Gemma 3 was vision-capable in some variants only); or Gemma 4 vision quality fails our evaluation set; or licensing changes at release.

Response. Swap the LLM provider via config. The LlmProvider abstraction (see 04-abstractions-and-contracts.md) makes this a one-line config change.

Fallback order:

1. Gemma 3 (vision-capable variant) — already tested in the prior prototype; known-working baseline. License (Gemma Terms) permits commercial use.
2. Qwen 2.5-VL (7B or larger) — Apache 2.0, fully permissive. Comparable quality on visual scene analysis. Has been a candidate in the original tech-stack docs.
3. LLaVA-Next — open weights, broad ecosystem support.
4. Specific newer models that have shipped since Jan 2026 — to be evaluated at integration time; not committed in this plan.

Evaluation gate. Before any model goes to production, it must pass our eval suite — a fixed set of ~20 scenario images with known expected outputs (smoke vs incense, fire vs reflection, crowd vs empty corridor, etc.). The eval suite lives in apps/ai-worker/eval/; new models run it as a CI step.

Time to switch. Hours — pull the new model image, update config, restart the deployment. Eval suite confirms within ~30 minutes that the new model meets the bar.

Cost. Negligible. Possibly a slight increase in GPU memory needs (some models are larger); plan for L4 24 GB minimum either way.

3. CNTXT / GCP Dammam account activation delay

Trigger. Account not active by the time M01 needs it (concretely: by the end of week 1 of work). Specific signals: CNTXT account manager not returning calls, billing not yet enabled, bank account verification stalled.

Response — staged workaround. Don’t wait. Proceed in a parallel temporary track:

1. For dev / build: spin up a temporary GCP project in a non-KSA region (the developer’s existing personal GCP account or a fresh one for the project). Use it strictly for development against synthetic data — never with real customer frames, never with anything from the Saudi Airlines site. The temporary project has its own budget caps and tight quotas, sized for dev (max ~$200/month).
2. For Tier 2 AI dev: Google AI Studio API for prompt engineering and model evaluation against the eval suite (synthetic / public images only). Free tier covers exploration; small paid usage if needed.
3. For test data: the open-source datasets and synthetic frames the AI engineer assembles for eval. No real surveillance video.

Production deployment is gated on the Dammam account being active. We do not ship customer-impacting workloads to a non-KSA region. Ever.

Communication. Melissa keeps CNTXT pressure on weekly. If activation hasn’t happened by M01 close, escalate.

Time. The temporary-project workaround is hours to set up. The wait for CNTXT is whatever it is.

Cost. Temporary project < $200/month. The activation timeline is the question.

4. Hardware procurement delay for the edge box

Trigger. The pilot’s edge box (the Linux server at the Saudi Airlines Arafat site) isn’t on-site by the time M04 / M09 needs it. Or it’s on-site but isn’t properly networked.

Response. Several layered fallbacks:

1. Use one of the spare desktop machines sitting in the Beam Wave office (per Melissa’s earlier note). Wipe Windows, install Ubuntu Server, run the bootstrap. ~1 hour of work.
2. VM on existing customer infrastructure — if Saudi Airlines has a Linux VM host with capacity at the site, deploy our software to a VM there. Same software stack; just runs in a VM instead of on bare metal. Requires customer IT cooperation.
3. Remote development against a placeholder — for the early milestones (M01–M03) we don’t need the real edge box. We can run the edge supervisor and edge-app containers on a developer’s machine pointed at the cloud. Defer the real edge install until M04 (camera ingestion) actually needs a building presence.

The 3D-modeler dependency. The 3D model is delivered by the same modeler who did the Clock Tower — assumed glb/gltf. If the format isn’t what we expect, the ThreeDModelLoader abstraction has fallback drivers (OBJ, manual extrusion from floor plans). We request the format we want before the model is built so this doesn’t surprise us at delivery.

5. Fire panel doesn’t speak a protocol we have a driver for

Trigger. Site survey reveals the FACP is a model we haven’t written a driver for. Common surprises: legacy panel with only contact-closure outputs; vendor-locked proprietary protocol; only-Modbus when we’d expected BACnet.

Response — three tiers of fallback:

1. Contact closure (always works). Every fire panel ever made has alarm-relay outputs — dry contacts that close when alarms fire. The edge box can detect these via a USB-to-GPIO adapter (e.g. NumatoLab GPIO board, ~$50). Our ContactClosureAdapter driver handles this; we get alarm-active / not-active signal with no zone granularity, but we still trigger the AI correlation pipeline on every alarm. This is the universal fallback.
2. Specific protocol driver. If the panel speaks BACnet, Modbus, or MQTT but we haven’t written a driver yet, we write the driver. Estimate: 1–2 days for BACnet/Modbus given existing libraries; longer for proprietary protocols.
3. Manual operator trigger (worst case). If we can’t talk to the panel at all by go-live, the dashboard exposes a “fire reported at zone X” manual trigger that operators can fire from the existing fire control room. The same AI correlation runs against the same cameras. Suboptimal but recoverable.

Site survey resolves this in week 1. A licensed BACnet/Modbus protocol-analyzer tool on the building’s network during the survey reveals what we’re dealing with.

6. NVR access denied or NVR-only camera access

Trigger. Site survey reveals cameras are not directly accessible from the network — they all route through an NVR — and the NVR vendor / access situation is locked down (no ONVIF Profile G, no credentials).

Response.

1. Try direct first. Many buildings have cameras both in an NVR and reachable directly. Try direct RTSP on each camera’s IP; many work.
2. Negotiate NVR access. With customer IT support, get credentials for the NVR via its native protocol (Hikvision ISAPI, Dahua RPC, Milestone MIP) or via ONVIF Profile G. Drivers exist for all of these in the NvrAdapter slot.
3. Replace cameras (worst case, costly). If neither direct nor NVR access works, the only fallback is to physically reroute or replace cameras. This is a 1–2 week disruption and outside our scope; we’d need to escalate to the customer + Al Baddad.

Mitigation: push hard for direct-access or NVR-with-credentials confirmation during site survey. Don’t accept “we’ll figure it out later.”

7. The 3D model is wrong, late, or unusable

Trigger. The modeler delivers a file that doesn’t load, has wrong scale, wrong axis convention, or arrives after M07 needs it.

Response.

1. Asset spec, given to the modeler in week 1. A short doc telling them exactly the format (glb preferred), the axis convention (Y-up, meters, North = +Z or whatever we standardize), the origin point, the naming convention for floors / rooms / zones, the level of detail expected. Sent before they start work. Saves the most pain at the back end.
2. ThreeDModelLoader driver fallbacks. If glb is delivered but malformed, the GlbLoader driver attempts repair. If a different format arrives, IfcLoader or ObjLoader slot in.
3. 2D extrusion fallback. If the 3D doesn’t arrive at all by M07, we fall back to extruding floor plans (image traced to polygons + per-floor height) into a basic 3D scene. Lower fidelity but operationally functional. The ManualExtrusionLoader driver exists for exactly this.
4. Static 3D fallback. Worst case: a clean simple box model of the building (right shape, no interior detail). The spatial-correlation logic still works because we map cameras and zones onto whatever 3D scene we have.

8. Schedule slip — what to drop first

If we’re tracking behind by mid-M07 and the Hajj window is closing in, we cut in this order:

1. First to cut: polish-only features in M08 (BI dashboard, secondary KPI panels, bilingual layout edge cases). The core operator + FD views are kept.
2. Second: Tier 2 LLM can degrade to Gemma 3 (already proven) without delay. If the cloud LLM is unavailable entirely (worst case), the platform still functions on the edge-side anomaly autoencoder alone — every anomaly-flagged frame becomes an operator-mediated review with no AI verdict attached. Less impressive, still operational; operators see “anomaly detected on camera X” instead of “smoke detected with 0.8 confidence.”
3. Third: digital-twin spatial correlation can degrade to “alert fires for all cameras on the same floor as the alarm zone” if PostGIS / 3D placement isn’t ready. Less surgical; still gets evidence to operators.
4. Fourth: per-camera calibration can degrade to a global anomaly threshold for all cameras initially. Lower precision; still useful.
5. Fifth (last resort): the alert-routing automation can be replaced by a Slack / Teams webhook that posts every alert to a channel the FD watches. Crude but immediate.

What we never cut:

• Authentication and RBAC. Even at minimum scope, the system must require login and respect tenant boundaries.
• Audit logging. Always on.
• Encryption (TLS, CMEK). Always on.
• Cost guardrails. Always on.
• Backups. Always on.

9. Schedule slip — what makes us re-plan instead of cut

Some slips are bigger than feature cuts and warrant a hard conversation with Melissa / Al Baddad / Saudi Airlines:

• CNTXT account never activates. No GCP Dammam = no production deployment. If this is unresolved by mid-M03, we have to either delay the pilot or find an alternative legal path (which is dubious for surveillance data).
• No cameras are reachable at the site (neither direct nor NVR). We can’t run an AI surveillance system without cameras. Need site IT intervention or equipment swap.
• The fire panel is genuinely unreachable. Contact closure is the universal fallback, but if even that is impossible (e.g. the panel is in a sealed cabinet we can’t open), the false-alarm-suppression feature isn’t deliverable. Pilot continues without it; we negotiate a contract adjustment.
• LOA gets signed with the wrong scope. This is the biggest risk identified in this entire project. The LOA-as-drafted has scope we are not building (elevators, access control, PA, on-prem GPU cluster, 100 TB storage server). If Beam Wave signs the LOA as-is without revising, we are contractually obligated to deliver hardware and integrations not in our actual plan, with liquidated damages at 0.5%/day. Melissa is aware. The LOA must be reconciled before signing.

10. The “we made it work, now what about scale” contingency

Less of a contingency than a planning note: if the pilot goes well and Saudi Airlines (or anyone else) wants to onboard more buildings fast, we are temporarily constrained by:

• YOLOv8 AGPL license — only relevant if YOLO returns as a parallel metadata feed (it was demoted out of the pilot critical path on 2026-05-13; see docs/plan/01-architecture-summary.md §0). Resolution path remains: buy Ultralytics commercial license, or swap to RT-DETR / YOLO-NAS / similar permissive model.
• CNTXT contract scope — verify our existing contract covers additional GCP resource consumption; new buildings = more cameras = more frames = more cost.
• Edge supervisor at scale — the M09 deliverable handles one building well. For dozens of buildings, we need the fleet-manager service (also in M09) to scale, which means moving its state out of in-memory into Redis or Postgres. Worth a session.
• 3D modeler capacity — one external modeler can do one building at a time. For Kingdom-wide rollout, we either onboard more modelers, automate exterior capture (drone photogrammetry), or simplify the digital-twin requirement for non-critical buildings.

None of these block the first pilot. All of them are real conversations 2–3 months out.

11. Operating principle for contingencies

When one of these triggers fires, the response is:

1. Note it in the relevant milestone’s Risks section (every milestone file has one).
2. Activate the prepared response above.
3. Communicate to the team and to Melissa within the day.
4. Update this document with what happened, what worked, what didn’t, and what we’d do differently. The document gets better with every contingency we actually hit.

Don’t panic. The architecture is built to absorb these.