Unattended encrypted boot (LUKS → TPM via Clevis)

How the edge box boots itself — encrypted disk and all — after a power loss in Arafat with nobody at the keyboard, and every hard-won lesson from making that actually work on Ubuntu.

This was proven end-to-end on edge-arafat-01 (the pilot box) in a controlled reboot: power-cycle → disk auto-unlocked from the TPM → no passphrase → key-only SSH back in ~18 s.

The goal

The box lives in Arafat. Power blips. There is no engineer on site. It must come back completely on its own: power on, decrypt its own root disk, boot, bring up SSH, start the supervisor, let the cloud find it. Every link in that chain has to be unattended or the whole thing is worthless.

The full chain:

power returns
  → BIOS "Restore on AC Power Loss = Power On"   (firmware, manual per box)
  → GRUB
  → initramfs: Clevis unseals the LUKS key from the TPM   (no passphrase)
  → OS boots
  → sshd up, key-only (password auth disabled)
  → wiqaia-edge-supervisor starts
  → cloud pings / finds / manages it

Miss any one link and a power blip turns into a truck roll.

The BIOS settings (manual, one-time, per box)

install.sh cannot set firmware portably (no BMC/IPMI on the pilot hardware), so this is a documented manual step in UEFI setup before the box ships. The principle, vendor-neutral:

Wake-on-LAN is a different feature (wake from sleep over the network) and is not what solves the power-loss case — the AC-power-restore setting is.

Pilot hardware: Dell Pro Tower Essential (QCT1255)

The Saudi Airlines pilot boxes are the Dell Pro Tower Essential (Dell’s 2025 rebrand of the entry-tier OptiPlex Tower), model QCT1255. Exact menu paths (BIOS updates may shuffle these slightly):

1. Enter BIOS — reboot, mash F2 at the Dell logo.
2. Power → AC Behavior → AC Recovery → Power On.
3. Power → Deep Sleep Control → Disabled. Dell-specific gotcha: leave Deep Sleep on and AC Recovery silently fails to wake the box. This one is non-obvious and not in the vendor-neutral table above.
4. Preboot Behavior → Warnings and Errors → Continue on Warnings and Errors. Leave Extend BIOS POST Time at 0.
5. Boot Configuration → Boot Sequence: internal NVMe enabled/top. Uncheck Enable PXE Boot Priority, Force PXE On Next Boot, Secure Digital (SD) Card Boot. Under Integrated Devices → USB Configuration, disable USB Boot Support for a field box.
6. Security → TPM 2.0 security: confirm TPM 2.0 Security On is enabled (it is — Clevis is unsealing), and that Clear is not set (default). The Clear toggle is what wipes the TPM seal.
7. Save & exit, reboot, confirm Clevis still unseals (no prompt).

Model-specific gotchas to know for the fleet:

• Firmware Device Tamper Detection (Security menu): if enabled and a tamper event fires (case opened / intrusion switch), the box logs the event and refuses to reboot until someone clears it at the BIOS level. Good security, surprising in the field. The intrusion switch is wired through BIOS — transporting or servicing a box will trip it; plan for someone at the console to clear the event on next boot.
• BIOS updates via fwupd work on Ubuntu (pushable over SSH for fleet management) — but a BIOS update can invalidate the TPM seal → Clevis can’t unseal → passphrase prompt. Never push firmware to a remote box without a recovery plan (this is the same operational rule as the caveats section below).
• Dell Command | Configure can script BIOS config across many QCT1255s once you’re past a handful of boxes.

As configured on edge-arafat-01 (pilot box)

Performed manually by Melissa on 2026-05-15, walked through with an assistant against the Dell QCT1255 manual:

• ✅ AC Recovery → Power On
• ✅ Deep Sleep Control → Disabled
• ✅ Warnings and Errors → Continue on Warnings and Errors
• ✅ TPM 2.0 sanity-checked (on; Clear not set)
• ⚠️ Boot order hardening NOT done; USB-boot state UNVERIFIED. Ubuntu/NVMe was already top so the Boot Sequence list was not touched. USB Boot Support / PXE / SD-card boot state was never checked — and that toggle lives on a different BIOS screen (Integrated Devices → USB Configuration) than the boot-order list that was reviewed, so its state is genuinely unknown, not assumed. Verify + harden before the box ships to Arafat. Tracked: Gitea #236 (label melissa-todo).

Security note on the open item. Because our Clevis TPM policy has no PCR binding (intentional — see below), the TPM will release the LUKS key to anything that boots on this machine and asks. If USB boot is enabled (unverified — that’s what #236 is to check), someone with physical access could boot foreign media and have the disk auto-unseal. This is exactly the “powers-on-THIS-box attacker” risk we consciously accepted for unattended resilience — disabling USB/PXE/SD boot is the defence-in-depth that narrows it, and **should be confirmed

• completed before this box leaves for a customer site**. It is not a blocker for the in-house pilot.

How it’s set up

All of it is in infra/edge-bootstrap/install.sh, which is idempotent and self-logging (/var/log/wiqaia-edge-install.log is your record).

Relevant flags (full list: install.sh --help):

• --host-prep-only — provision the host (Docker, service user, unit, binary, keypair, SSH lockdown, LUKS→TPM) but skip cloud registration + supervisor start. For prepping a box before the cloud exists. Re-run without this flag once cloud is up to register — earlier steps no-op.
• --binary-file <path> — install a locally-built supervisor instead of fetching the (pre-cloud non-existent) binary mirror.
• --skip-luks — run host-prep as one clean pass and do the TPM enroll separately/deliberately later.
• --luks-only — run only the LUKS→TPM enroll, nothing else. Run it interactively (ssh -t) so you’re prompted for the passphrase, or set LUKS_PASSPHRASE.

The pilot sequence we actually ran (cloud not up yet):

# 1. host prep, no LUKS yet, local binary, no cloud
sudo ./install.sh --host-prep-only --skip-luks 
     --binary-file /tmp/wiqaia-edge-supervisor

# 2. LUKS → TPM, interactive so the passphrase is typed straight into
#    the tool (never env / argv / logs / chat). Run from your terminal:
ssh -t rizoma@<box> 'cd ~/wiqaia/infra/edge-bootstrap && sudo ./install.sh --luks-only'

# 3. controlled reboot test (see "Proving it" below)

The passphrase is never passed through automation by default. With LUKS_PASSPHRASE set it’s read once from the env; otherwise Clevis prompts on the TTY. It is never written to disk, argv, or the log.

Why Clevis, and NOT bare systemd-cryptenroll

This is the single most important lesson. The “obvious” approach — systemd-cryptenroll --tpm2-device=auto + tpm2-device=auto in /etc/crypttab — enrolls fine but never unlocks at boot on Ubuntu.

Ubuntu’s default initrd is the script-based initramfs-tools cryptroot hook. It does not understand crypttab’s tpm2-device=auto (that’s a systemd-cryptsetup feature, and Ubuntu’s stock initrd is not systemd-driven for this path). So the systemd-cryptenroll token sits in the LUKS header, is never consumed, and the box just falls back to the passphrase prompt — looking exactly like a “wrong” setup when the enrollment was actually perfect.

Clevis (clevis, clevis-luks, clevis-initramfs, clevis-tpm2) ships a real initramfs-tools hook + boot script that performs the TPM unseal during that script-based boot. That is the supported Ubuntu + initramfs-tools answer. enroll_luks_tpm() in install.sh uses Clevis exclusively now.

If you ever see “enrollment succeeded but it still asks for the passphrase at boot” — this is why. Check clevis luks list and that the clevis hook is actually in the initrd (not just that a TPM token exists).

The three traps that cost real time

Document these so nobody re-loses the hours:

1. libtss2-rc is missing on a stock Ubuntu

systemd-cryptenroll --tpm2-device=list reports “TPM2 support is not installed” even though /dev/tpmrm0 exists and the kernel TPM is perfectly fine. systemd dlopens libtss2-esys/-rc/-mu/-tcti-device; Ubuntu’s base ships every one except libtss2-rc. The fix is to install the full stack — clevis-tpm2 (and tpm2-tools) depend on it, so the Clevis path fixes this automatically. install.sh installs the Clevis stack before binding.

2. lsinitramfs | grep -q under set -o pipefail — false negative

This one cost ~an hour and looked like flaky, non-deterministic behaviour (“works when I run it by hand, fails in the script”).

lsinitramfs lists thousands of lines. grep -q exits on the first match and closes the pipe. lsinitramfs, still writing, gets SIGPIPE → exit 141. Under set -o pipefail the pipeline’s status is that 141, so a check like:

if lsinitramfs /boot/initrd.img-$(uname -r) | grep -q "usr/bin/clevis$"; then

reports failure even though clevis IS in the initrd. Manual one-liners (no pipefail) pass, which is exactly why it looked nondeterministic.

Rule (enforced by a regression test): never pipe a large producer into grep -q in a pipefail script. Capture to a temp file (or herestring for small output) and grep that:

tmpls=$(mktemp)
lsinitramfs "/boot/initrd.img-$(uname -r)" >"$tmpls" 2>/dev/null || true
grep -q "usr/bin/clevis$" "$tmpls"; rc=$?
rm -f "$tmpls"

tests/test-install-script.mjs has a guard that fails the build if lsinitramfs|grep or clevis luks list|grep ever reappears.

3. The clevis-initramfs post-install can race the first rebuild

The first update-initramfs triggered during the clevis apt install can miss the hook. install.sh rebuilds all kernels (-u -k all, removing running-vs-latest ambiguity), verifies the running kernel’s initrd specifically, retries once, and hard-fails if clevis still isn’t there. It never claims success blind — the verify is the gate.

No PCR binding — deliberate, and the tradeoff

The Clevis pin is bound with no PCR ids (tpm2 '{}'), i.e. TPM-presence only. This is intentional and was explicitly approved:

• Why no PCRs: binding to firmware/kernel PCRs (0/4/8/9) would brick unattended boot on every kernel auto-update — fatal for a remote box nobody can attend.
• What we keep: the drive is useless pulled out of this box (theft at rest is covered).
• What we consciously trade away: protection against an attacker who powers on this box. Correct trade for an unattended remote appliance — you cannot have a human type a passphrase after every power blip 900 km away.

pcr_ids must never be added — there’s a regression test for that too.

Proving it (the only real test)

TPM/LUKS unlock cannot be unit- or container-tested (no TPM, no encrypted root in CI). The only proof is a controlled reboot, and the only safe place to do it for the first time is in-house, before the box ships — never first-time on a box already in Arafat.

Procedure:

1. Be physically at the box with the LUKS passphrase in hand.
2. Reboot. Watch the screen.
3. Pass: it returns on the network in ~1–2 min with no passphrase prompt. (boot_id changed = real reboot; SSH answers; root is the LUKS-backed LV; systemd-cryptsetup@dm_crypt-0 “Finished” in the journal.)
4. Fail-safe: if it’s not back in ~3 min, the screen shows the normal passphrase prompt. Type it (the passphrase keyslot is never removed — guaranteed recovery), box boots, diagnose, retry. Zero data risk; worst case is “type the passphrase once, like today”.

Verify the setup without rebooting:

# TPM token + that the passphrase slot is still there (≥2 keyslots):
sudo cryptsetup luksDump /dev/nvme0n1p3 | grep -E 'Keyslots:|luks2|Tokens:|clevis'
sudo clevis luks list -d /dev/nvme0n1p3            # expect a 'tpm2' line
# clevis actually in the initrd (NOT via `| grep -q` — see trap #2):
tmp=$(mktemp); sudo lsinitramfs /boot/initrd.img-$(uname -r) >"$tmp"; 
  grep -E 'usr/bin/clevis$|scripts/local-top/clevis' "$tmp"; rm -f "$tmp"

Operational caveats — the “never do this remotely” list

Auto-unlock can only stop working on deliberate hardware/firmware events, never a normal reboot/power-cycle/kernel update (we bound no PCRs precisely so updates are safe). The ones that do break it:

• BIOS/firmware update or “Clear TPM” in setup → the seal is gone → next boot drops to the passphrase prompt → and auto-power-on does not save you (it powers on and waits at the prompt). Never push firmware/BIOS changes to a remote box without a plan for someone to enter the passphrase.
• Mainboard / TPM swap, or TPM disabled in BIOS → same: passphrase prompt.
• Disk moved to another machine → won’t unseal. That’s the security property working as designed.

In every case: no data loss, LUKS header intact, passphrase slot intact. Consequence is only “a human must type the passphrase once”.

Therefore:

• Keep the LUKS passphrase escrowed (password manager — not the repo, not chat). It is the permanent break-glass.
• Never remove the passphrase keyslot to “tidy up”. As long as it exists there is no lockout, only inconvenience. install.sh only ever adds a keyslot; it never wipes one (regression-tested).

Recovery (TPM cleared / box moved)

1. Boot with the passphrase at the console (recovery slot — always works).
2. Re-enroll: sudo ./install.sh --luks-only (idempotent — if a clevis tpm2 binding already exists it skips the bind and just rebuilds + verifies the initrd; if the TPM was cleared it re-binds, prompting for the passphrase).
3. Controlled reboot test again.

SSH lockdown (done in the same host-prep)

harden_ssh() disables password auth entirely (key-only) and pins cloud-init so a reboot/datasource refresh can’t turn it back on:

• Drop-in is /etc/ssh/sshd_config.d/00-wiqaia-hardening.conf — the 00- prefix is deliberate: sshd uses the first value seen, so it must sort before 50-cloud-init.conf or cloud-init wins.
• /etc/cloud/cloud.cfg.d/99-wiqaia-ssh.cfg sets ssh_pwauth: false.
• Validated with sshd -t, then reloaded not restarted so the live session never drops. Always confirm a fresh key-only connection on a second SSH before trusting it — we will not lock ourselves out.

Repeatability / end state

Today it’s a hybrid only because the cloud (binary mirror + fleet manager) isn’t up yet. Once it is, the wrapper collapses into the documented one-liner from infra/edge-bootstrap/README.md:

curl -fsSL https://install.wiqaia.sa/edge | sudo bash -s --   --token <one-time-token> --building-id <uuid>

It stays a parameterised one-liner (per-box token + building-id, and LUKS_PASSPHRASE once for an encrypted box) — that’s inherent to per-box identity, not a gap. The BIOS settings remain the one genuinely-manual per-box step.

Testing

• Static (pnpm -F @wiqaia/edge-bootstrap test, no Docker): bash syntax, every flag/step, the SSH-lockdown ordering, the no-wipe / no-PCR / no-|grep safety invariants.
• Container e2e (pnpm -F @wiqaia/edge-bootstrap test:docker): runs install.sh in a fresh Ubuntu container, twice, proving idempotency + host-prep-only + the hardening drop-in. The LUKS step self-skips with no /etc/crypttab (no encrypted root in a container), so the container stays green and fast — TPM/LUKS coverage is the reboot test, by necessity.
• Run the container gate before any real-box mutation. It caught a pre-existing harness bug (the canonical systemd unit was never staged into the build context — test:docker now stages it).

See also: Edge host preparation, Supervisor won’t start.