M10 — Hardening and UAT Prep

One-line goal: the platform survives intentional failures, the runbooks tell whoever’s on call exactly what to do, KPI targets are verified against a recorded test set, training is delivered to the customer’s operators, and the on-prem-GPU contingency has been exercised end-to-end.

After M10, we are ready for UAT sign-off and for the Hajj operational window.

Tracks involved

• Cloud Infra — primary on DR, backups, runbooks.
• AI Worker — KPI verification, eval-suite regression, accuracy/latency benchmarks.
• All tracks — runbooks for their respective surfaces.

Dependencies

• M01 through M09 complete. M10 is the polish-and-prove phase that depends on everything else being real.

Deliverables

1. Observability dashboards

Production-grade dashboards in Cloud Monitoring:

• System health overview — every service’s up/down indicator, latency p50/p95/p99, error rate, queue depths.
• AI pipeline performance — frames per minute through each tier, end-to-end latency, escalation rate, false-alarm-correlation success rate, GPU utilization.
• Alert lifecycle — alerts created, acknowledged, resolved per hour, mean time to ack/resolve, alerts by severity.
• Cost overview — already in M01, now refined with real running data.
• Tenant scorecard — per-building KPIs (the customer-facing operational dashboard).
• Audit-log integrity — chain verification status, recent integrity-check results.

Each dashboard linked from the runbook for the corresponding domain.

2. Alerts (operational, not safety)

• Cloud Monitoring alert policies for:
  • Service down > 2 minutes → page on-call.
  • API error rate > 5% sustained 5 minutes → page on-call.
  • GPU node fails to scale within 5 minutes of queue spike → warn.
  • Edge gateway offline > 5 minutes → warn.
  • Audit-log chain integrity check failed → page immediately.
  • Backup job failed → page within 1 hour.
  • Cost threshold breached → 25/50/75/90/100/110/120% budget alerts paging humans (M01); automated kill switch deferred for the pilot, see decisions.
• Alert routing: Slack channel + email + Melissa’s phone for criticals.

3. Runbooks

docs/runbooks/ with one file per operational concern:

• incident-response.md — what to do when an alert fires (per category).
• disaster-recovery.md — when the primary Cloud SQL instance dies, when the GKE cluster has an outage, when Dammam region has issues.
• edge-bootstrap.md — onboarding a new edge gateway from scratch.
• edge-troubleshooting.md — common edge faults (offline, cert expired, container crash-loop).
• camera-troubleshooting.md — camera offline, wrong credentials, ONVIF quirks.
• fire-panel-troubleshooting.md — adapter not connecting, zone mapping wrong.
• llm-troubleshooting.md — Tier 2 latency, parse failures, model swap procedure.
• backup-and-restore.md — verified backup procedure with restore drill notes.
• cost-emergency.md — 120% budget breach response.
• security-incident.md — suspected breach response, key rotation, customer notification.
• onboarding-new-customer.md — building registration, edge install, camera placement, calibration kickoff.
• offboarding-customer.md — data deletion, key revocation, audit-trail preservation.

Each runbook follows a consistent template: Symptoms · Likely causes · Immediate response · Diagnostic commands · Escalation · Post-incident review.

4. Backup and restore drill

• DR drill scripted in infra/dr/drill.sh — kills the primary Cloud SQL, promotes the regional replica, verifies the API survives, generates a report.
• First drill executed manually; report saved in docs/runbooks/disaster-recovery.md with timings.
• Drill run on a monthly cadence during the pilot; quarterly post-pilot.
• Restore-from-backup drill: restore last night’s backup to a sibling DB; query for known-seed data; confirm.

5. On-prem-GPU contingency dry run

Even if the pilot uses cloud GPUs as planned, we exercise the on-prem path during M10:

• Stand up the spare GPU workstation that was pre-ordered as insurance.
• Install our edge supervisor + AI-worker containers configured for WIQAIA_DEPLOYMENT_MODE=on_prem.
• Run the eval suite — confirm verdicts match the cloud version within tolerance.
• Run a synthetic alarm-correlation flow end-to-end — confirm the SLO is met from on-prem.
• Document the procedure in docs/runbooks/on-prem-deployment.md so it’s a 1-day procedure, not a 1-week panic, if we ever need it.

6. KPI verification against recorded test set

The customer’s UAT criteria include accuracy and latency targets per the contract (and per the architecture vision §17.2). Verify:

• AI accuracy — eval-suite pass rate at ≥ 90% (per M06).
• End-to-end latency — alarm trigger → evidence on operator screen, p95 < 30 s, p99 < 60 s.
• Live-video latency — click to first frame, p95 < 2 s.
• Camera-offline detection — within 60 s.
• System availability — synthetic uptime monitoring over the pilot phase (target 99.5% during Hajj window).
• Audit-log integrity — chain verification passing every day.

Each KPI captured as a documented test, run, with results stored in docs/runbooks/kpi-baseline.md for ongoing comparison.

7. Penetration test prep

• Internal pre-pen-test sweep:
  • Verify all OWASP Top 10 mitigations from 03-data-protection.md.
  • Confirm no service exposes a public IP it shouldn’t.
  • Confirm Sentry / Cloud Logging / BigQuery have no plaintext customer data.
  • Run automated scanners (Trivy on images, ZAP on the web app, sqlmap-style probes on the API).
• External pen test scheduled with an NCA-authorized provider for ~1 month post-pilot launch (per 03-data-protection.md §13).

8. Operator training materials

• docs/training/operator-handbook.md — comprehensive guide for Civil Defence operators: how to log in, what each view does, how to interpret AI verdicts, how to escalate, how to mark false alarms, who to contact for help.
• docs/training/fd-handbook.md — equivalent for fire-department dispatchers.
• docs/training/admin-handbook.md — for building admins: how to onboard cameras, configure rules, manage users.
• Quick-reference cards — single-page printouts for the operator’s station with the most common actions.
• Recorded walkthrough videos for each handbook (screen-capture with voiceover, EN + AR).

9. Live training session

• Scheduled with Saudi Airlines + Civil Defence representatives.
• Hands-on with the actual pilot system.
• Comprehension check at the end — short scenario quiz, signed off as completed.
• Recording archived for new-operator onboarding later.

10. Final acceptance prep

• All milestones from M01–M09 verified, sign-offs in docs/plan/COMPLETION_LOG.md.
• UAT script (the operator + FD scripts from M08) re-run as a fresh-eyes acceptance test.
• Customer (or stand-in) signs the Acceptance Test Report.
• Final acceptance pack prepared for the contractual milestone: architecture overview, runbooks, KPI baseline, training records, audit-trail summary, IP-statement reaffirmation.

Verification

1. DR drill succeeds. Kill the primary Cloud SQL via the drill script; API survives with < 2 minutes of degraded responses; replica promoted; all writes accepted; rollback to primary clean.
2. Backup restore succeeds. Restore last night’s backup to a sibling instance; query for canonical seed data; confirm correctness; restore completes within 15 min.
3. On-prem GPU mode runs eval suite green. All eval-suite cases pass on the on-prem workstation; verdicts match cloud-mode within tolerance.
4. All KPI targets pass on a recorded test set. Documented in docs/runbooks/kpi-baseline.md; signed off by Melissa.
5. All runbooks tested. Each runbook has been executed at least once (by the team running through the documented steps) and refined based on the actual experience.
6. Pen test pre-sweep clean. Automated scanners report no critical / serious vulnerabilities. Manual review of high-risk endpoints documented.
7. Operator training delivered. Saudi Airlines + Civil Defence representatives have completed training and signed off on comprehension.
8. Audit-log integrity verified continuously. Daily chain check has passed every day for the last 14 days at the time of UAT.
9. Cost trajectory matches forecast. Cumulative pilot spend through M10 is within 20% of the forecast in 02-cost-guardrails.md §7.
10. No P1 / P2 bugs open. Bug tracker shows zero open critical or high-severity bugs at UAT time.

Risks

Open questions

• Customer-side acceptance signatory. Who signs the Acceptance Test Report on behalf of Saudi Airlines / Al Baddad? Confirm before UAT.
• Pen test provider selection. Multiple NCA-authorized providers exist; selection is a 1-day procurement. Recommend before M10.
• Hajj on-call rotation. Who’s on call during May 25 – June 5? A rotation between Melissa, the cloud engineer hire, the AI engineer hire, and the Saudi-side hands-on person. Document in docs/runbooks/on-call.md before Hajj starts. No single point of human failure.

Exit criteria

All 10 verification items pass. Customer acceptance signature on the Acceptance Test Report. Final acceptance pack assembled and delivered. M10 sign-off entry in docs/plan/COMPLETION_LOG.md.

This is the milestone that closes the pilot delivery.