Store-and-forward queue is full

The on-box dashboard shows queue depth approaching the configured capacity (default 5 GB), or you see QueueFull errors in the supervisor’s logs. The store-and-forward queue holds events the edge couldn’t send to the cloud — when it fills, the next event is refused rather than silently dropping a higher-severity event that’s already in the queue.

This is the platform working correctly under prolonged cloud unreachability. The fixes depend on whether the queue is full of noise or signal.

0. Confirm

wiqaia-edge status
sudo journalctl -u wiqaia-edge-supervisor --since "1 hour ago" | grep -i 'queuefull|store_forward'

If you see repeated QueueFull errors, walk the steps below.

1. Why isn’t the queue draining?

The queue drains automatically when the tunnel to the cloud reconnects. If it isn’t draining, the cloud is unreachable — see edge-supervisor-wont-start runbook section 3 (“tunnel dial failed”).

A re-established tunnel will drain at the configured per-event rate. Don’t restart the supervisor unless the runbook tells you to — that just delays the drain by a few seconds and adds a journald log entry.

2. How full is “full”? What’s in it?

Today, the supervisor doesn’t expose a direct CLI shortcut for queue stats (it’s an open issue). For now:

sudo sqlite3 /var/lib/wiqaia/edge/store_forward.db 
    "SELECT topic, severity, COUNT(*) FROM events GROUP BY topic, severity ORDER BY severity DESC, topic"

You’ll see rows like:

analysis-events|3|1502
frame-events|0|824631
telemetry|0|11240

Severity 0 = INFO; 1 = LOW; 2 = MODERATE; 3 = HIGH; 4 = CRITICAL.

3. The queue is mostly INFO/LOW (noise)

This is the normal failure mode — Tier-0 motion frames piling up because the cloud’s unreachable. Once the tunnel reconnects, the queue drains; the severity-aware overflow already kicked the oldest INFO/LOW events out to make room for newer ones, so no signal was lost.

No action required. Wait for the tunnel to reconnect; monitor the depth dropping.

If you genuinely need to free space NOW (rare; usually a disk-pressure issue elsewhere):

# Stop the supervisor so it isn't writing while we touch the DB.
sudo systemctl stop wiqaia-edge-supervisor
# Drop events older than 1 hour AND severity ≤ MODERATE.
sudo sqlite3 /var/lib/wiqaia/edge/store_forward.db 
    "DELETE FROM events WHERE severity <= 2 AND enqueued_at < strftime('%s', 'now') - 3600"
sudo systemctl start wiqaia-edge-supervisor

Don’t ever DELETE severity ≥ 3 manually — those are the events the queue was designed to protect.

4. The queue is full of HIGH/CRITICAL

This is the rare-and-scary case. Something at the customer site is producing genuine signal that the cloud isn’t seeing.

Do not delete these. They are the audit trail the customer (and Civil Defence) will want to reconstruct what happened during the outage.

Steps:

  1. Confirm the tunnel is actually down (see step 1).
  2. Get the tunnel back up — that’s the entire fix.
  3. After drain completes, capture the count of escalations during the outage window for the post-incident report: 
    sudo sqlite3 /var/lib/wiqaia/edge/store_forward.db 
    "SELECT MIN(enqueued_at), MAX(enqueued_at), COUNT(*) FROM events WHERE severity >= 3"

If the tunnel CAN’T be brought back up within a few hours and the disk is genuinely about to fill, escalate to the on-call engineer. We’d rather sacrifice older INFO frames than miss new CRITICAL events; the eviction policy does this automatically.

5. Disk full from queue growth

Default queue cap is 5 GB. If df -h /var/lib/wiqaia shows the partition over 90 %, the cap may be wrong for the host.

Reconfigure via the supervisor’s environment (edit the systemd unit):

sudo systemctl edit wiqaia-edge-supervisor
# In the editor, add:
#   [Service]
#   Environment="EDGE_QUEUE_CAPACITY_BYTES=2147483648"
sudo systemctl restart wiqaia-edge-supervisor

(Note: the supervisor doesn’t yet read this env var; it’s a documented hook for the future PR that wires the queue config through internal/config. Tracked separately.)

Monitoring

The on-box dashboard at http://<edge-host>:8081 (M09 #127) is the canonical view. The cloud-side fleet dashboard (post-M09) will roll up queue depth across all gateways so an operations engineer sees the whole fleet at a glance.

WiqAIa+ Docs · Rizoma