fix(beefcake/impermanence): boot the ephemeral root fully — sops initrd identity + /var parent perms #727

Merged

lytedev merged 4 commits from beefcake-impermanence-boot-fixes into main 2026-07-09 03:49:54 -05:00

Conversation 0 Commits 4 Files changed 5 +267 -5

lytedev commented 2026-07-08 17:31:03 -05:00

Owner

Copy link

Second wave of fixes for the beefcake impermanence flip. Machine-id (#726) got PID1 up; this gets the service graph up, adds a VM regression guard, and makes the host identity recoverable. Validated in place on the live impermanent root (outage tolerated — pebble catching mail): after these fixes beefcake reached systemctl is-system-running == running, 0 failed units, on the ephemeral root — first fully-working flip.

Commits

1. fix: boot the ephemeral root fully — two independent root causes:
   • sops age identity absent in initrd. sops-install-secrets runs in initrd activation and sops-nix's key paths don't follow services.openssh.hostKeys — they default to hardcoded /etc/ssh/* (absent on @blank that early) + /var/lib/sops-nix/key.txt (stage-2 bind, never persisted). → /run/secrets empty → knot/stalwart/kanidm/… down. Fix: point sops at the persisted ed25519 key (readable in initrd; /persist is stage-1 neededForBoot). Drop the RSA-GPG path.
   • /var, /var/lib, /var/cache came up 0700 → non-root service users can't traverse into /var to reach their StateDirectory. impermanence propagates the /persist source-parent mode, and /persist/var{,/lib} were 0700. Fix: tmpfiles d /var 0755 (+ lib, cache) re-asserts 0755 after local-fs, before services — self-corrects any from-scratch /persist.
2. docs: runbook — fix the -- --boot footgun (was passing --boot to nix build = silent live-switch) → deploy --boot … --hostname 192.168.0.9 (LAN, not VPN — headscale). Adds Part 6: Retry #3 (the reboot test) with the no-downgrade pre-flight.
3. test: VM regression guard — extends the rollback prototype to reproduce the /var-perms bug in a clean-room VM (non-root service + persisted /var/lib dir + forced 0700 /persist source) and assert the fix. Passed on dragon: "all 0755 despite 0700 /persist source", "varprobe (non-root) wrote + kept its marker through the wipe".
4. feat: back up host identity keys — beefcake's ed25519 (= the sops identity) + rsa ssh host keys were install-random disk-only state; captured into secrets/beefcake/host-identity.yml encrypted to the master key so identity is recoverable. Roundtrip-verified. NOT a runtime sops.secrets entry (circular — the key is what sops needs to decrypt) → inert to build/reboot; recovery is an off-host, master-key, pre-first-boot lay-down (documented in the runbook).

Version safety

main + config-only diff — no flake.lock change. Builds 26.05.20260623.667d5cf, the exact nixpkgs beefcake runs now. Not a downgrade, not cross-release — config + initrd change only, no toolchain re-exec to wedge on. Reboot required (initrd change), but no live activation (--boot).

Not reboot-stable until deployed

The live host runs the old flip closure with these fixes hand-applied. Retry #3 = deploy this --boot + reboot (Daniel-driven, LAN, iDRAC visibility). Clean-boot assertions: ls /run/secrets | wc -l (~44), stat -c '%a' /var /var/lib (755), is-system-running (running). Merge follows the reboot proof (verify-before-merge).

Remaining follow-up (its own PR)

Optionally decouple the sops identity from the ssh host key onto a dedicated intentional age key (add-recipient-without-remove + updatekeys, ssh-derived key kept as fallback). Deferred — not needed for a working impermanent boot.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

Second wave of fixes for the beefcake impermanence flip. Machine-id (#726) got PID1 up; this gets the **service graph** up, adds a VM regression guard, and makes the host identity recoverable. Validated in place on the live impermanent root (outage tolerated — pebble catching mail): after these fixes beefcake reached `systemctl is-system-running == running`, **0 failed units**, on the ephemeral root — first fully-working flip. ## Commits 1. **fix: boot the ephemeral root fully** — two independent root causes: - **sops age identity absent in initrd.** `sops-install-secrets` runs in *initrd* activation and sops-nix's key paths don't follow `services.openssh.hostKeys` — they default to hardcoded `/etc/ssh/*` (absent on `@blank` that early) + `/var/lib/sops-nix/key.txt` (stage-2 bind, never persisted). → `/run/secrets` empty → knot/stalwart/kanidm/… down. Fix: point sops at the **persisted** ed25519 key (readable in initrd; `/persist` is stage-1 `neededForBoot`). Drop the RSA-GPG path. - **`/var`, `/var/lib`, `/var/cache` came up `0700`** → non-root service users can't *traverse* into `/var` to reach their `StateDirectory`. impermanence propagates the `/persist` source-parent mode, and `/persist/var{,/lib}` were `0700`. Fix: tmpfiles `d /var 0755` (+ lib, cache) re-asserts `0755` after `local-fs`, before services — self-corrects any from-scratch `/persist`. 2. **docs: runbook** — fix the `-- --boot` footgun (was passing `--boot` to `nix build` = silent live-switch) → `deploy --boot … --hostname 192.168.0.9` (LAN, not VPN — headscale). Adds **Part 6: Retry #3** (the reboot test) with the no-downgrade pre-flight. 3. **test: VM regression guard** — extends the rollback prototype to reproduce the `/var`-perms bug in a clean-room VM (non-root service + persisted `/var/lib` dir + forced `0700` `/persist` source) and assert the fix. **Passed on dragon:** *"all 0755 despite 0700 /persist source"*, *"varprobe (non-root) wrote + kept its marker through the wipe"*. 4. **feat: back up host identity keys** — beefcake's ed25519 (= the sops identity) + rsa ssh host keys were install-random disk-only state; captured into `secrets/beefcake/host-identity.yml` encrypted to the master key so identity is recoverable. Roundtrip-verified. NOT a runtime `sops.secrets` entry (circular — the key is what sops needs to decrypt) → inert to build/reboot; recovery is an off-host, master-key, pre-first-boot lay-down (documented in the runbook). ## Version safety `main` + config-only diff — no `flake.lock` change. Builds `26.05.20260623.667d5cf`, the exact nixpkgs beefcake runs now. **Not a downgrade, not cross-release** — config + initrd change only, no toolchain re-exec to wedge on. Reboot required (initrd change), but no live activation (`--boot`). ## Not reboot-stable until deployed The live host runs the old flip closure with these fixes hand-applied. Retry #3 = deploy this `--boot` + reboot (Daniel-driven, LAN, iDRAC visibility). Clean-boot assertions: `ls /run/secrets | wc -l` (~44), `stat -c '%a' /var /var/lib` (755), `is-system-running` (running). Merge follows the reboot proof (verify-before-merge). ## Remaining follow-up (its own PR) Optionally decouple the sops identity from the ssh host key onto a dedicated intentional age key (add-recipient-without-remove + `updatekeys`, ssh-derived key kept as fallback). Deferred — not needed for a working impermanent boot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

lytedev added 1 commit 2026-07-08 17:31:03 -05:00

fix(beefcake/impermanence): boot the ephemeral root fully — sops initrd identity + /var parent perms ... 9a8905345d

Flip attempt #2 (2026-07-08) booted past the machine-id fix (#726) but left
~14 services down. Two independent root causes, both found by validating in
place on the live impermanent root and both fixed here; beefcake then reached
`systemctl is-system-running == running` with 0 failed units on the ephemeral
root.

1. sops age identity absent in initrd. sops-install-secrets runs in *initrd*
   activation and sops-nix's key paths do NOT follow services.openssh.hostKeys
   — they default to hardcoded /etc/ssh/* (absent on the @blank root that
   early) and age.keyFile=/var/lib/sops-nix/key.txt (a stage-2 bind, never
   actually persisted). sops found no identity, tried the RSA key as GPG, and
   failed "0 successful groups" -> /run/secrets empty -> knot/stalwart/kanidm/...
   down. Fix: point sops at the PERSISTED ed25519 host key, readable in initrd
   because /persist is a stage-1 neededForBoot mount (it derives the
   sshd-at-beefcake age recipient on every beefcake secret). Drop the RSA-GPG
   path.

2. /var, /var/lib, /var/cache came up mode 0700 on the ephemeral root, which
   blocks every non-root service user from *traversing* into /var to reach its
   StateDirectory (knot/caddy/kanidm/forgejo/mosquitto/otel all "permission
   denied" on mkdir/chdir of /var/...). Masked at first boot by cause #1;
   surfaced once secrets resolved. impermanence creates the bind-mount parents
   by propagating the mode of the matching source under /persist, and
   /persist/var{,/lib} had been created 0700. Fix: tmpfiles `d /var 0755` (+
   /var/lib, /var/cache) re-asserts 0755 after local-fs and before services,
   every boot, regardless of the /persist source mode. The live /persist
   parents were also chmod'd 0755.

FOLLOW-UP (Daniel): provision the ed25519 host key declaratively (encrypted
in-repo to the master key, laid on /persist at setup) so the host identity is
intentional + recoverable, not lucky install-random state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

lytedev referenced this pull request from a commit 2026-07-08 17:45:11 -05:00

docs(beefcake/impermanence): fix --boot flag placement + add Retry #3 reboot-test runbook

lytedev added 1 commit 2026-07-08 17:45:11 -05:00

docs(beefcake/impermanence): fix --boot flag placement + add Retry #3 reboot-test runbook ... b00104fb33

Part 3 step 2 had `deploy ... -- --boot`, which passes --boot to nix build
(silent live-switch, wedge risk) instead of deploy-rs. Correct form is
`deploy --boot` before `--`, over the LAN (--hostname 192.168.0.9; a VPN
deploy severs headscale). Adds Part 6: the retry-#3 procedure for the
already-flipped host — no-downgrade pre-flight (nixpkgs_4 667d5cf == what
beefcake runs, so config/initrd-only), --boot deploy + reboot, and the three
clean-boot assertions #727 proves (/run/secrets populated, /var 0755,
is-system-running).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

lytedev added 1 commit 2026-07-08 19:04:28 -05:00

test(beefcake/impermanence): VM regression guard for /var parent-perms ... a34275bf14

Extend the rollback prototype to reproduce flip-#2's SECOND root cause in a
clean-room VM: environment.persistence of a /var/lib dir owned by a non-root
user, with the /persist source parents forced to 0700 in initrd (mirroring
beefcake). The systemd.tmpfiles.rules fix must re-assert 0755 on the ephemeral
/var,/var/lib,/var/cache so the non-root varprobe service can traverse to its
persisted state dir. rollback-demo asserts all three parents are 0755 and the
marker was written + survived the wipe.

Proven on dragon: 'all 0755 despite 0700 /persist source — fix works',
'varprobe (non-root) wrote + kept its marker through the wipe'. Guards the
class the deploy-gate still can't (no /persist boot-convergence tier).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

lytedev added 1 commit 2026-07-08 19:10:15 -05:00

feat(beefcake/impermanence): back up host identity keys to repo (recoverable) ... a23ac646a5

beefcake's ed25519 + rsa ssh host keys were install-random state that only
existed on disk; the ed25519 key IS the sops age identity (derives
age1etv56f...=sshd-at-beefcake). Capture both, base64'd, into
secrets/beefcake/host-identity.yml encrypted to the master key (age &daniel)
so the host identity is intentional + recoverable rather than lucky disk-only
state. Roundtrip-verified (ed25519 priv<->pub matches; derived recipient ==
sshd-at-beefcake).

Deliberately NOT a runtime sops.secrets entry: the ed25519 key is the identity
sops needs to decrypt everything, so laying it via sops is circular. It's a
bootstrap/recovery artifact — decrypt off-host with the master key and place on
a fresh /persist before first boot. Runbook gains a 'Recovery: host identity'
section. Inert to the build + reboot (no secret is laid from it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G7HHrn1sxz4ozvREyHmi4i

lytedev referenced this pull request 2026-07-08 19:31:03 -05:00

fix(beefcake/lite): eval-derive dummy-secret keys (gate fixture) #728

lytedev referenced this pull request from a commit 2026-07-08 19:37:36 -05:00

docs(beefcake/impermanence): fix --boot flag placement + add Retry #3 reboot-test runbook

lytedev force-pushed beefcake-impermanence-boot-fixes from a23ac646a5 to 26487ec2f9 2026-07-08 19:37:36 -05:00 Compare

lytedev merged commit 26487ec2f9 into main 2026-07-09 03:49:54 -05:00

lytedev deleted branch beefcake-impermanence-boot-fixes 2026-07-09 03:49:55 -05:00

lytedev referenced this pull request 2026-07-09 09:36:21 -05:00

feat(beefcake/mass): open TCP 8927 (Sendspin) for MA companion-app players #731

lytedev referenced this pull request from a commit 2026-07-09 17:18:32 -05:00

fix(beefcake-host): three cutover-fatal boot gaps caught in pre-flight (#9-#11)

lytedev referenced this pull request from a commit 2026-07-10 11:24:50 -05:00

fix(beefcake-host): three cutover-fatal boot gaps caught in pre-flight (#9-#11)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is wrong

  

enhancement

New feature

  

question

Awaiting information

  

scary

Potentially problematic change

  

upstream

Awaiting upstream changes

No labels

bug

enhancement

question

scary

upstream

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lytedev/nix!727

No description provided.