M06 — Tier 2 LLM Analysis

Architecture revision (2026-05-13): in the 2-tier architecture, the Tier-2 multimodal LLM is the primary safety classifier, not a second opinion behind a YOLO gate. It runs on a single always-warm L4 GPU in me-central2 and analyzes every frame the edge escalates — both anomaly-driven escalations and the baseline sampling cadence. See docs/plan/01-architecture-summary.md §0.

One-line goal: frames escalated from the edge (anomaly-flagged, baseline samples, or alarm-correlated) are analyzed by a multimodal LLM hosted in Dammam, which returns structured threat assessments (threat_detected, severity, confidence, reasoning, false_alarm_likelihood, person_count, detected_objects) that the alert engine uses to drive operator decisions.

This is the signature AI capability — the thing that lets the platform say “yes, that’s smoke from incense, not a fire” with evidence.

Tracks involved

• AI Worker — primary. LLM serving, prompt template system, LlmProvider driver implementation, evaluation suite.
• Backend — Tier2AnalysisEvent persistence, escalation pipeline, the analysis-events ↔ tier2-events plumbing.
• Cloud Infra — gpu-tier2 node pool (L4 GPU with enough VRAM for the chosen model), CMEK on the model bucket, careful network egress controls.
• Frontend — Tier 2 verdict display on alerts, evidence-package rendering.

Dependencies

• M05 complete (edge is escalating frames to the analysis-events topic — anomaly-driven, baseline cadence, and alarm-correlated).
• M03 (escalation_record, prompt_template tables exist).
• 04-abstractions-and-contracts.md §5.5 — LlmProvider interface defined.

Deliverables

1. Tier 2 LLM serving

apps/ai-worker/tier2/ as a Python FastAPI service deployed on the gpu-tier2 node pool. Uses vLLM for high-throughput serving.

1.1 Model selection — Gemma 4 with verified fallbacks

• Primary: Gemma 4 multimodal vision model. Before integration, verify it ships with vision capability (some Gemma generations have been text-only initially). If not vision-capable at integration time, fall back automatically per the contingency plan.
• Fallback 1: Gemma 3 vision variant (already-tested baseline from the prototype).
• Fallback 2: Qwen 2.5-VL 7B (Apache 2.0, fully permissive).
• Selection by config flag WIQAIA_LLM_PROVIDER=gemma_4|gemma_3|qwen_25_vl.

All three implement the same LlmProvider interface; switching is a config + container-image change, no code change.

1.2 Hosting topology

• Models stored in GCS (Dammam, CMEK), pulled to local SSD on pod start.
• One always-warm L4 GPU (g2-standard-8) hosting one vLLM pod. minReplicas: 1, maxReplicas: 2 (max 2 to allow a brief surge during a model swap or canary). NOT scale-to-zero. Fires happen year-round and the platform’s fire-detection guarantee is incompatible with a 4-7 minute GPU cold-start.
• vLLM batches concurrent requests; sustained throughput target on one L4 is 5-7 inferences/sec with batching. At 40 cameras × baseline cadence (1 frame per 30 s per camera) the steady-state load is ~1.3 escalations/sec — well within budget — plus anomaly-driven escalations on top.

1.3 Inference

Per FrameEscalation event arriving on analysis-events from the edge:

1. Decrypt the frame payload (application-layer encryption from the edge — see 03-data-protection.md §5).
2. Render the prompt from the active prompt template (prompt_template_id), substituting in: camera location, time-of-day, zone description, escalation reason (anomaly / baseline_sample / alarm_correlated), any active alarm context.
3. Run multimodal inference at temperature 0.1, max-tokens 500, response-format JSON.
4. Parse output against the canonical schema (threat_detected, threat_type, severity, confidence, reasoning, recommended_action, false_alarm_likelihood, person_count, detected_objects).
5. If parse fails (model emitted invalid JSON), retry once with a “STRICT JSON” reminder; if still failing, log + return threat_detected=true, severity=UNKNOWN, reasoning="LLM output unparseable — operator should review" (fail safe).
6. Publish Tier2AnalysisEvent to the tier2-events topic and persist the canonical AnalysisEvent row (this is the single source of truth — in the 2-tier architecture there is no separate Tier-1 AnalysisEvent).
7. Persist EscalationRecord row in Postgres with token counts for cost accounting.

2. Prompt template system

2.1 Versioned templates

Templates stored as rows in prompt_template per M03. Each template has:

• template_id (canonical name like smoke_verify_v2_1)
• version (SemVer)
• scenario_type (general / smoke_verify / crowd_assess / restricted_zone / night_anomaly / fire_alarm_correlation)
• system_prompt (the LLM persona / safety-rules preamble)
• instructions (the per-scenario question and JSON output schema)
• response_schema (strict JSON Schema; output is validated against it)
• is_active (only one per scenario_type at a time)

2.2 Initial template set (M06 ships these)

• general_safety_v1 — the default catch-all. “Examine this frame from a building camera. Report any fire, smoke, unauthorized person, crowd hazard, fall, intrusion, or other safety concern. Account for common false-alarm sources in this region: bakhoor incense, prayer smoke, steam from kitchens, reflections, authorized cleaning crews.”
• smoke_verify_v1 — fires when an alarm panel signals smoke or fire. “A fire alarm was triggered in this zone. Examine this camera that has line-of-sight to the alarm zone. Distinguish between actual fire/smoke and likely false-alarm causes (bakhoor, cooking steam, dust). Report findings.”
• crowd_assess_v1 — for crowd density alerts. “This zone has a current occupancy of {count}/{max}. Is the crowd safely distributed, dangerously concentrated, or showing signs of panic?”
• restricted_zone_v1 — for detections in zones marked is_restricted = true. “This area is normally restricted to authorized personnel. Are the people visible authorized (e.g. cleaning crew, security staff in uniform), or is this an unauthorized entry?”
• night_anomaly_v1 — for high-anomaly-score frames captured between 22:00 and 05:00. “Nighttime anomaly detected. Report what’s visible and whether it warrants human attention.”
• fire_alarm_correlation_v1 — invoked by M07 when a FACP event triggers spatial correlation. Combines smoke_verify_v1 with explicit alarm-zone context.

Each template is a Postgres row; new versions appended; the active one used at inference time.

2.3 Template editing UI

Admin-only screen at /admin/prompt-templates:

• List templates by scenario_type.
• Side-by-side diff between active and proposed.
• Run-against-eval-set button (described in §3.3).
• Activate / deactivate.

All edits audit-logged.

3. Evaluation suite

apps/ai-worker/eval/ — the gate every new model or prompt template passes before going to production.

3.1 Eval set composition

~30 scenarios, mix of:

• Smoke / fire positives (real smoke, real flames, varied light/angles): ~6
• Smoke false positives (bakhoor, prayer, steam, cooking, dust, fog, reflections, contrails through windows): ~8
• Crowd hazards (dense crowds, single-direction flow blockage, fallen person in crowd): ~4
• Restricted-zone scenarios (uniform staff, unauthorized civilian, ambiguous): ~4
• Empty-corridor null cases (model should report nothing): ~4
• Edge cases (low light, motion blur, partial occlusion, off-angle): ~4

Each scenario has:

• Input frame
• Context (zone description, alarm state, etc.) matching the prompt template’s expectations
• Expected output (threat_detected, threat_type, allowed range for confidence and false_alarm_likelihood)
• Tolerance band (the LLM is non-deterministic; the eval allows reasonable variation)

3.2 Eval execution

• Runs as a CI job on every PR that touches apps/ai-worker/ or prompt_template migrations.
• Pass criteria: ≥ 90% of cases pass, no critical-severity case (fire / smoke positive) fails.
• Results posted to the PR; failing PRs cannot merge.

3.3 Production promotion

• New models / templates run the eval offline against the current production-active version.
• If new beats current on ≥ 80% of cases without critical regressions, the new version is promoted to 50% of escalations for 1 week (canary).
• Operator-feedback signal (confirm vs dismiss vs false_alarm marking) compared during canary; if new continues to beat current, promote to 100%; otherwise rollback.

4. False-alarm verdict UI

When an alert is rendered (M07 + M08), the Tier 2 verdict is displayed prominently:

• Verdict badge: REAL FIRE / PROBABLE FALSE ALARM / NEEDS HUMAN REVIEW (color-coded).
• Confidence percentage (LLM’s confidence × (1 - false_alarm_likelihood)).
• Reasoning text (the LLM’s free-form reasoning field, shown verbatim, EN/AR translated post-hoc if needed).
• Recommended action.
• Per-camera evidence cards (each camera’s frame snapshot + its individual verdict if multiple cameras analyzed).

Operator can override with one click (“Confirm fire”, “Mark as false alarm”). Override action is fed back as training signal for future prompt-template iteration.

5. Per-tenant cost accounting

Each EscalationRecord includes input_tokens and output_tokens. Daily / monthly per-tenant inference cost is computed and surfaced on the cost dashboard. If a tenant disproportionately burns LLM budget, an alert lets us investigate before the budget cap fires.

6. Rate limiting

• Per-building escalation rate cap (default: 60 escalations per minute per building) to prevent a misbehaving edge or a malicious actor from flooding Tier 2.
• Per-camera escalation cap (default: 4 escalations per minute per camera).
• Over-cap escalations queue for up to 60 s; if still over after queuing, drop with a “rate_limited” audit log entry.

7. Observability

• LLM-specific dashboard: tokens per second, time-to-first-token p50/p95/p99, queue depth, escalation rate per building, false-alarm-likelihood distribution.
• Alert on: average inference time > 10 s, parse failure rate > 5%, queue depth > 100 frames for > 2 min.

Verification

1. Smoke + fire positive case escalates correctly. Feed the model a frame with visible smoke; verdict is REAL FIRE or NEEDS REVIEW with confidence > 0.7, reasoning mentions smoke specifically.
2. Incense / bakhoor false-positive case is identified. Feed a frame with visible bakhoor smoke and a context indicating “alarm triggered”; verdict is PROBABLE FALSE ALARM with false_alarm_likelihood > 0.7, reasoning mentions incense.
3. Empty corridor case returns no threat. Feed a frame of an empty corridor; verdict is no threat, confidence in “no threat” > 0.8.
4. Eval suite passes with the chosen model. Full eval suite executes; ≥ 90% pass; zero critical regressions.
5. Model swap works. Switch WIQAIA_LLM_PROVIDER from gemma_4 to qwen_25_vl; deployment cycles; the next escalation is served by Qwen; results are still structured JSON conforming to the schema.
6. Prompt template versioning works. Create smoke_verify_v2; run eval; activate; next smoke_verify-scenario escalation uses v2.
7. JSON-parse failure handled. Force the model to emit invalid JSON (e.g., temporarily increase temperature); the retry-with-reminder kicks in; if still failing, the fail-safe NEEDS_HUMAN_REVIEW response is published.
8. Rate limit fires correctly. Stress test with 200 escalations/minute on one building; over-cap escalations are rate-limited and audit-logged.
9. End-to-end latency under 10 s. From edge FrameEscalation publish to cloud Tier2AnalysisEvent publish, p95 < 10 s under steady-state load. (No cold-start case — the pod is always warm.)
10. Cost accounting reflects reality. Inference token usage matches per-tenant accounting; daily totals match GPU node-hours billed.

Risks

Open questions

• Gemma 4 vision availability confirmation. Verify at integration time. If not yet, default to Gemma 3.
• Prompt template language. EN throughout for the model itself (most models are English-strongest); the operator-facing translation of reasoning is a post-processing step. Decision: render LLM output in English; UI translates on the client side using a small translation pass (Google Translate API in Dammam? — pending cost / sovereignty assessment) or operator views with a toggle.
• How aggressively to cache escalation results. If the same frame is escalated twice within seconds, do we cache the verdict? Probably yes to save cost; verify with a hashing key.

Exit criteria

All 10 verification items pass. Eval suite passing at the chosen-model + chosen-prompt versions. Cost per escalation measured and below budget. M06 sign-off entry in docs/plan/COMPLETION_LOG.md.