fix(battleship): declaratively ensure guest persist zvol before domain start #760

Merged

lytedev merged 1 commit from guest-persist-zvol-ensure into main

2026-07-13 16:01:20 -05:00

lytedev commented

2026-07-13 15:55:13 -05:00

Owner

What

Declaratively guarantee the beefcake guest's vdb persist zvol (rpool/beefcake-persist) exists before the libvirt domain is defined/started, so a host cold-boot can never again fail on a missing backing disk.

Why (the outage this fixes)

The guest domain (packages/hosts/beefcake/slot-domain.nix) attaches two disks: vda → /dev/zvol/rpool/beefcake-blue (the real guest OS+data) and vdb → /dev/zvol/rpool/beefcake-persist (forward-plumbing for the unfinished Phase-4 shared persist). rpool/beefcake-persist was never created, so the first true cold define+start — a host reboot on 2026-07-13 — failed with Cannot access storage file '/dev/zvol/rpool/beefcake-persist' and took the guest (all of beefcake's services) down.

Prior reboots survived only because libvirt did a managed-save resume, which restores a running domain and does not re-validate disks. A cold define+start validates every backing path, so the missing zvol stayed hidden until a genuinely cold boot. vdb is vestigial from the guest's point of view (it imports its rpool from vda only) but is still declared in the domain XML, so the host must provide it. Recovery was an imperative zfs create -s -V 100G rpool/beefcake-persist + nixvirt restart — no data loss.

The fix

New systemd.services.ensure-guest-persist-zvol oneshot (Type=oneshot, RemainAfterExit=true) in packages/hosts/battleship.nix:
- create-if-absent only — zfs create -s -V 100G rpool/beefcake-persist (sparse, 100 GiB, matching the size created during recovery so there's no divergence with the live zvol). It never destroys/recreates an existing zvol and never touches beefcake-blue.
- waits (bounded ~15s) for the /dev/zvol/... node to appear, erroring if it never does.
- ordering: After=zfs.target zfs-import.target, Before=nixvirt.service libvirtd.service, WantedBy=multi-user.target. Both libvirtd.service (autostarts domains during its own startup) and nixvirt.service (After=Requires=libvirtd.service, applies the declarative domains) can start the guest, so it's ordered before both.
Promotes persistZvolDataset to a top-level let binding so the dataset name and /dev/zvol path share a single source of truth.
Not disko: disko is install-time/destructive and dangerous against the live host rpool holding guest data, and it does not reconcile a missing zvol at boot. Full disko-ification of the host pool is deferred future work (noted in-file).

Runbook (`lib/doc/beefcake-guest-hugepages.md`)

Adds an "Operational guards / lessons" section:

Pre-flight before ANY host reboot: virsh domblklist beefcake-blue and confirm every /dev/zvol/... backing path exists (cold start validates all disks; managed-save resume does not).
EFI default-generation gotcha: systemd-boot honors an EFI LoaderEntryDefault/LoaderEntryOneShot var over loader.conf's default; fix with bootctl set-oneshot <gen>.conf then bootctl set-default ''.
Notes the new oneshot guards guard #1's failure mode for the persist zvol specifically.

Validation

nix fmt clean.
nix eval .#nixosConfigurations.battleship.config.system.build.toplevel.drvPath succeeds.
Confirmed the built unit's directives: After=zfs.target zfs-import.target, Before=nixvirt.service libvirtd.service, WantedBy=multi-user.target, Type=oneshot, RemainAfterExit=true.

Applies on the next host activation — no reboot needed, since the zvol already exists live and the oneshot is a no-op there.

Not deployed, not merged.

## What Declaratively guarantee the beefcake guest's `vdb` persist zvol (`rpool/beefcake-persist`) exists **before** the libvirt domain is defined/started, so a host cold-boot can never again fail on a missing backing disk. ## Why (the outage this fixes) The guest domain (`packages/hosts/beefcake/slot-domain.nix`) attaches two disks: `vda` → `/dev/zvol/rpool/beefcake-blue` (the real guest OS+data) and `vdb` → `/dev/zvol/rpool/beefcake-persist` (forward-plumbing for the unfinished Phase-4 shared persist). **`rpool/beefcake-persist` was never created**, so the first true cold `define+start` — a host reboot on 2026-07-13 — failed with `Cannot access storage file '/dev/zvol/rpool/beefcake-persist'` and took the guest (all of beefcake's services) down. Prior reboots survived only because libvirt did a managed-save **resume**, which restores a running domain and does **not** re-validate disks. A cold `define+start` validates every backing path, so the missing zvol stayed hidden until a genuinely cold boot. `vdb` is vestigial from the guest's point of view (it imports its rpool from vda only) but is still **declared** in the domain XML, so the host must provide it. Recovery was an imperative `zfs create -s -V 100G rpool/beefcake-persist` + nixvirt restart — no data loss. ## The fix - New `systemd.services.ensure-guest-persist-zvol` oneshot (`Type=oneshot`, `RemainAfterExit=true`) in `packages/hosts/battleship.nix`: - **create-if-absent only** — `zfs create -s -V 100G rpool/beefcake-persist` (sparse, 100 GiB, matching the size created during recovery so there's no divergence with the live zvol). It **never** destroys/recreates an existing zvol and never touches `beefcake-blue`. - waits (bounded ~15s) for the `/dev/zvol/...` node to appear, erroring if it never does. - ordering: `After=zfs.target zfs-import.target`, `Before=nixvirt.service libvirtd.service`, `WantedBy=multi-user.target`. Both `libvirtd.service` (autostarts domains during its own startup) and `nixvirt.service` (`After=Requires=libvirtd.service`, applies the declarative domains) can start the guest, so it's ordered before both. - Promotes `persistZvolDataset` to a top-level `let` binding so the dataset name and `/dev/zvol` path share a single source of truth. - **Not disko:** disko is install-time/destructive and dangerous against the live host rpool holding guest data, and it does not reconcile a missing zvol at boot. Full disko-ification of the host pool is deferred future work (noted in-file). ## Runbook (`lib/doc/beefcake-guest-hugepages.md`) Adds an "Operational guards / lessons" section: 1. Pre-flight before ANY host reboot: `virsh domblklist beefcake-blue` and confirm every `/dev/zvol/...` backing path exists (cold start validates all disks; managed-save resume does not). 2. EFI default-generation gotcha: systemd-boot honors an EFI `LoaderEntryDefault`/`LoaderEntryOneShot` var over loader.conf's `default`; fix with `bootctl set-oneshot <gen>.conf` then `bootctl set-default ''`. 3. Notes the new oneshot guards guard #1's failure mode for the persist zvol specifically. ## Validation - `nix fmt` clean. - `nix eval .#nixosConfigurations.battleship.config.system.build.toplevel.drvPath` succeeds. - Confirmed the built unit's directives: `After=zfs.target zfs-import.target`, `Before=nixvirt.service libvirtd.service`, `WantedBy=multi-user.target`, `Type=oneshot`, `RemainAfterExit=true`. Applies on the next host activation — **no reboot needed**, since the zvol already exists live and the oneshot is a no-op there. Not deployed, not merged.

lytedev added 1 commit

2026-07-13 15:55:15 -05:00

fix(battleship): declaratively ensure guest persist zvol before domain start

/ check-format (push) Successful in 9s

Details

/ build (push) Has been skipped

Details

999ee33277

The beefcake guest domain (slot-domain.nix) declares TWO disks: vda
(/dev/zvol/rpool/beefcake-blue, the real guest OS+data) and vdb
(/dev/zvol/rpool/beefcake-persist, forward-plumbing for the unfinished
Phase-4 shared persist). rpool/beefcake-persist was never created, so
the first true cold define+start (a host reboot on 2026-07-13) failed
with "Cannot access storage file .../beefcake-persist" and took the
guest — all of beefcake's services — down. Prior reboots survived only
because libvirt did a managed-save *resume*, which does NOT re-validate
disks; a cold define+start validates every backing path, so the missing
zvol stayed hidden until a genuinely cold boot. vdb is vestigial from
the guest's view (it imports its rpool from vda only) but is still
DECLARED, so the host must provide it.

Add an ensure-guest-persist-zvol oneshot that idempotently creates the
persist zvol (create-if-absent only; -s -V 100G matching the size
created imperatively during recovery; never destroys/recreates and never
touches beefcake-blue), waits for its /dev/zvol node, and is ordered
after zfs.target/zfs-import.target and before libvirtd.service and
nixvirt.service (both can start the guest). Promote persistZvolDataset
to a top-level let binding so the dataset name and device path have a
single source of truth.

NOT disko: disko is install-time/destructive and would be dangerous
against the live host rpool that holds guest data, and it does not
reconcile a missing zvol at boot. A full disko-ification of the host
pool is deferred future work (noted in-file).

Applies on the next host activation (no reboot needed — the zvol already
exists live, so the oneshot is a no-op there). Also documents two
operational guards in lib/doc/beefcake-guest-hugepages.md: a
domblklist pre-flight before host reboots, and the stale-EFI-var
default-generation gotcha.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011m7LpmJkaSpr9NiKvaatNv

lytedev merged commit 999ee33277 into main

2026-07-13 16:01:20 -05:00

lytedev deleted branch guest-persist-zvol-ensure

2026-07-13 16:01:23 -05:00