M09 — Edge Supervisor and Auto-Update

One-line goal: a fresh Ubuntu box, given the bootstrap one-liner + a one-time token, comes online, registers with the cloud, pulls and runs all edge containers; pushing a new container version from the cloud triggers a blue/green update with auto-rollback on health-check failure — and the same on the next building, with no per-site intervention.

After M09, onboarding a new building takes hours instead of days, and updating the entire fleet is a single deploy from the cloud.

Tracks involved

• Edge — primary. Go supervisor, mTLS tunnel, container lifecycle, install script.
• Cloud Infra — fleet-manager service, cloud-side certificate authority, deployment pipeline for edge images.
• Backend — fleet-manager API (registration, version pinning, health rollup), audit-log integration for edge updates.

Dependencies

• M01 (Go supervisor scaffolding, install-script skeleton, Artifact Registry).
• M02 (auth — edge registration produces a service-account identity).
• M04+ (the edge containers exist and have stable container-image identities).

Deliverables

1. Edge supervisor — Go service

apps/edge-supervisor/ — Go 1.22, builds to a single static binary wiqaia-edge-supervisor (~10 MB).

1.1 Lifecycle

• Runs as systemd unit wiqaia-edge.service (Type=notify, restart=always).
• Reads config from /etc/wiqaia/edge.yaml (paths, registry, log level).
• Reads bootstrap state from /var/lib/wiqaia/state/ (mTLS certs, registration token, last-known fleet config).
• Talks to local Docker daemon via Unix socket.
• Self-restarts on failure; if it crash-loops, systemd backs off; a watchdog timer pings the cloud about supervisor-down via a separate path (HTTP webhook).

1.2 Capabilities

• Tunnel: persistent outbound-only mTLS to https://edge.wiqaia.sa/ (or internal-mesh equivalent). gRPC bidi-streaming for command + telemetry traffic; reconnect with backoff.
• Container management: pulls images from Artifact Registry using the device’s service-account identity; runs them via Docker API; tracks health via Docker health-checks + supervisor’s own probes.
• Update orchestration: receives DeployCommand from cloud; for each declared container, pulls the new image, starts it alongside the old one (blue/green), health-checks for a configurable window, then either promotes (stop old) or rolls back (stop new).
• Telemetry: emits EdgeTelemetryEvent every 60 seconds with the supervisor’s view of CPU/mem/disk, container statuses, camera counts, queue depths.
• Local config syndication: receives building / camera / panel configuration over the tunnel; persists to /var/lib/wiqaia/state/ so an offline edge still has the last-known config.
• Local web server: serves apps/edge-dashboard/ on http://<edge-ip>:8443/ over LAN with a self-signed cert (browser warning accepted by operators on local network). Auth via the cloud Keycloak with cookies issued by the edge supervisor’s local auth proxy.

2. The bootstrap install script

infra/edge-bootstrap/install.sh and hosted at https://install.wiqaia.sa/edge.

curl -fsSL https://install.wiqaia.sa/edge | sudo bash -s -- --token <onetime-token>

The script:

1. Verifies Ubuntu 22.04 (or 24.04 — confirm whether LTS is current at pilot time) on x86_64.
2. Updates apt and installs Docker (from Docker’s official repo, not the Ubuntu snap).
3. Adds the user to the docker group.
4. Downloads the supervisor binary from https://install.wiqaia.sa/edge/binaries/wiqaia-edge-supervisor-v{version}-linux-amd64 (with checksum verification).
5. Drops the systemd unit file.
6. Generates a key pair locally; sends the public key + the one-time token to the cloud /v1/edge/register endpoint.
7. Cloud responds with a signed mTLS certificate + a fleet-manager URL.
8. Supervisor starts, connects, pulls initial container set, reports back.
9. Done — total ~10 minutes on a reasonable internet connection.

The one-time token is generated in the admin UI when an operator creates a new edge gateway record. Single-use, time-limited (24 hours).

3. Cloud fleet manager

apps/api/src/fleet-manager/ — a NestJS module that owns:

• Edge registration: validates one-time tokens, issues mTLS certs (via an internal CA), creates the edge_gateway row in Postgres.
• Tunnel termination: accepts gRPC streams from edges (via a separate gRPC ingress, not the main REST API); routes commands and telemetry.
• Version pinning: each edge gateway has a target version per container, settable via API or UI; deviations trigger a deployment.
• Health rollup: aggregates EdgeTelemetryEvent and surfaces fleet-wide health (single edge for the pilot, but the pattern scales).
• Certificate rotation: 90-day cycle, automated, with at-least-one-week overlap.

4. Container deployment workflow

4.1 Image promotion

• CI builds + pushes images to Artifact Registry with tags like me-central2-docker.pkg.dev/wiqaia-pilot-prod/wiqaia/edge-camera-manager:v1.2.3.
• A promote-edge-image workflow tags the image as stable, canary, or pinned-to-<gateway-id> for staged rollouts.

4.2 Deployment trigger

• An admin selects a target version per container in the fleet-manager UI (or sets a fleet-wide default).
• The fleet manager publishes a DeployCommand per affected edge.
• Each edge runs the blue/green dance:
  1. Pull new image.
  2. Start a new container alongside the old one with health-check enabled.
  3. Wait for healthy probe (configurable, default 60 s).
  4. Switch traffic / restart MediaMTX bindings (the supervisor handles port/socket reassignment).
  5. Stop the old container.
  6. Report success.
• On health-check failure: stop the new container, leave the old one running, report failure, surface in the UI.

4.3 Canary deployment

For risk-aware rollouts:

• Promote new image to canary on a single edge (or 10% of fleet for larger deployments).
• Wait 24 hours.
• If no issues, promote to stable.

For the pilot (one edge), canary = wait an hour after the first deploy of any change.

5. Supervisor self-update

The supervisor itself is the one piece that can’t update via Docker — it’s the thing running Docker. Two layers:

• Default: distributed as a .deb package from our own apt repo (also hosted at install.wiqaia.sa). unattended-upgrades pulls updates nightly during a configurable maintenance window. The package’s postinst script restarts systemd cleanly.
• Emergency: the supervisor checks for a SupervisorUpdateCommand from the cloud; if it receives one with a valid signed package URL + signature, it downloads, verifies, and triggers dpkg -i with restart. Used only for security-critical fixes.

6. Store-and-forward queue

apps/edge-app/store_forward/ — SQLite-backed queue inside one of the edge containers (logically owned by the supervisor; physically a sibling container).

• Persists outbound events (FrameEvent, AnalysisEvent, AlarmEvent, AuditLogEntry, EdgeTelemetryEvent) when the cloud tunnel is unhealthy.
• Drains in chronological order when connectivity restores.
• Configurable cap (default: 24 hours of events at the typical rate). When full, oldest events with lowest severity drop first; high-severity events are protected.
• Self-heals: on supervisor restart, rebuilds in-memory state from SQLite.

7. Edge local dashboard

apps/edge-dashboard/ (same React + Vite SPA as the main dashboard, configured for the local edge API):

• Camera grid showing only this building’s cameras.
• Local alert log.
• AI status (Tier 0 motion + anomaly autoencoder active on edge; Tier 2 LLM mode — cloud or on-prem).
• “Configuration drift” indicator (this edge’s container versions vs. what the cloud says it should be running).
• Accessible only on the building’s LAN.

8. Operational tooling

• wiqaia-edge-cli — a small CLI on the edge box for diagnostics. status, logs <container>, restart <container>, health-check, re-register. Strictly read-only for production scenarios where possible; mutations audit-logged via the supervisor.
• Runbooks in docs/runbooks/edge-*.md: install, register, common-faults, escalation paths.

Verification

1. Fresh-box install. A clean Ubuntu 22.04 VM, given the bootstrap one-liner with a valid token, becomes a registered, online edge gateway with all its expected containers running. Total time < 15 minutes.
2. Container deploy + rollback. Push a new edge-camera-manager version with an intentional health-check failure; the supervisor pulls, attempts the swap, detects unhealthy, rolls back to the previous version. Cloud sees the rollback report.
3. Successful upgrade. Push a clean new version; supervisor performs the blue/green swap; cameras keep streaming without interruption (verify by watching a live feed during the swap).
4. Cloud disconnect resilience. Sever the edge’s internet connection for 1 hour; cameras continue to view locally; events queue; on reconnect, all queued events drain to the cloud in order.
5. Certificate rotation. Manually trigger cert rotation; supervisor receives new cert, swaps without disconnection, reports rotation in audit log.
6. Telemetry visible. Within 60 seconds of supervisor start, the fleet manager UI shows the edge as online with current container versions, CPU/mem/disk metrics.
7. Supervisor self-update. Apt-repo update triggers; unattended-upgrades installs; systemd restarts; supervisor reconnects without manual intervention.
8. Local dashboard works. From a laptop on the same LAN as the edge box, the local dashboard loads and shows live cameras.
9. Configuration drift detected. Manually docker stop one of the edge containers; supervisor restarts it; if a stop is persistent (e.g. wrong image manually run), the drift indicator surfaces in the cloud fleet view within 1 minute.
10. CLI diagnostics work. wiqaia-edge-cli status returns a useful one-screen summary. wiqaia-edge-cli logs camera-manager --tail 100 works.

Risks

Open questions

• Edge box hardware spec confirmation. The “desktop machines in the Beam Wave office” likely have plenty of CPU/RAM for a single building’s load (a brand-new desktop with 16GB+ RAM is fine). Confirm specifics; document minimum.
• Whether to support Docker Compose or pure Docker API. Pure Docker API is simpler; Docker Compose adds a wrinkle for engineers used to it. Default: Docker API (the supervisor implements the orchestration logic itself).
• Customer IT preference for unattended upgrades. Some customers disable auto-updates by policy. Default to enabled; allow per-deployment override.

Exit criteria

All 10 verification items pass on the actual pilot edge box (or a representative VM). Bootstrap install completed on a fresh machine to prove the path. M09 sign-off entry in docs/plan/COMPLETION_LOG.md.