feat(beefcake): move cold persist state off the SSD onto zstorage #801

Merged

lytedev merged 1 commit from cold-persist into main

2026-07-21 13:13:36 -05:00

lytedev commented

2026-07-21 12:31:18 -05:00

Owner

Precise fix for the SSD pressure that paused the guest on 2026-07-21: move the cold state off the SSD, keep the hot state on it.

What the audit actually found

The guest 30.6G /persist is mostly not data worth keeping on SSD:

path	size	nature
`var/cache/restic-backups-*`	15.0G	systemd `CacheDirectory`, rebuildable
`root/.cache`	5.1G	cache
`var/lib/jellyfin`	3.8G	regenerable metadata/artwork
`var/log`	1.7G	journal + caddy
forgejo (repos+db+indexers)	1.3G	the actually-hot set

Change

Move the restic caches + jellyfin metadata (~19G) onto /storage (virtiofs -> zstorage, 2.4T free). They still survive reboots, preserving the original intent (the restic caches were persisted deliberately so post-reboot runs do not re-fetch indexes from the remote sftp repos).

forgejo deliberately stays on SSD. virtiofs is ~430x slower for git (see forgejo.nix) and PR #756 fixed 2-7s repo pages by getting git off it. Moving /persist wholesale to zstorage would have re-broken that while leaving the caches on SSD -- exactly backwards.

Why explicit fileSystems binds, not a second persistence root

impermanence emits bind units with DefaultDependencies=false and no ordering against /storage. At boot they could bind before the virtiofs mount exists, and services would silently write to the ephemeral root and lose data at the next reboot. Using fileSystems + depends = [ "/storage" ] makes it explicit -- verified in the generated fstab:

/storage/beefcake-cold/var/lib/jellyfin /var/lib/jellyfin none x-systemd.requires-mounts-for=/storage,bind,x-gvfs-hide 0 0

⚠️ NOT a plain deploy -- migrate first

Deploying alone would present empty dirs to jellyfin/restic (data still sits in /persist). Before/with the deploy, on the guest:

systemctl stop jellyfin restic-backups-local restic-backups-rascal restic-backups-benland
mkdir -p /storage/beefcake-cold/var/lib /storage/beefcake-cold/var/cache
rsync -aHAX --info=progress2 /persist/var/lib/jellyfin/        /storage/beefcake-cold/var/lib/jellyfin/
rsync -aHAX --info=progress2 /persist/var/cache/restic-backups-local/   /storage/beefcake-cold/var/cache/restic-backups-local/
# ...rascal, benland likewise

then deploy, confirm the binds are live (findmnt /var/lib/jellyfin), verify jellyfin + a restic run, and only then remove the old copies from /persist to actually reclaim the SSD.

Restic caches are rebuildable, so if you would rather not rsync 15G over virtiofs you can simply let restic repopulate them (first run after is slower).

Deferred deliberately

/root/.cache (5.1G) nests inside the wholesale-persisted /root.
/var/log (1.7G) -- capping journald SystemMaxUse is the better lever.

Verified

Config evaluates; fstab carries the ordering option on all four binds; tmpfiles creates the backing dirs. Not deployed -- given I caused the outage earlier today, I did not want to ship an impermanence change to this host unvalidated.

🤖 Generated with Claude Code

https://claude.ai/code/session_015dwmWkDcFN7CnZ5TmLK9nY

Precise fix for the SSD pressure that paused the guest on 2026-07-21: move the **cold** state off the SSD, keep the **hot** state on it. ## What the audit actually found The guest 30.6G `/persist` is mostly not data worth keeping on SSD: | path | size | nature | | --- | --- | --- | | `var/cache/restic-backups-*` | **15.0G** | systemd `CacheDirectory`, rebuildable | | `root/.cache` | 5.1G | cache | | `var/lib/jellyfin` | 3.8G | regenerable metadata/artwork | | `var/log` | 1.7G | journal + caddy | | **forgejo** (repos+db+indexers) | **1.3G** | the actually-hot set | ## Change Move the restic caches + jellyfin metadata (**~19G**) onto `/storage` (virtiofs -> zstorage, 2.4T free). They still survive reboots, preserving the original intent (the restic caches were persisted deliberately so post-reboot runs do not re-fetch indexes from the remote sftp repos). **forgejo deliberately stays on SSD.** virtiofs is ~430x slower for git (see `forgejo.nix`) and PR #756 fixed 2-7s repo pages by getting git off it. Moving `/persist` wholesale to zstorage would have re-broken that while leaving the caches on SSD -- exactly backwards. ## Why explicit fileSystems binds, not a second persistence root impermanence emits bind units with `DefaultDependencies=false` and **no ordering against `/storage`**. At boot they could bind before the virtiofs mount exists, and services would silently write to the **ephemeral** root and lose data at the next reboot. Using `fileSystems` + `depends = [ "/storage" ]` makes it explicit -- verified in the generated fstab: ``` /storage/beefcake-cold/var/lib/jellyfin /var/lib/jellyfin none x-systemd.requires-mounts-for=/storage,bind,x-gvfs-hide 0 0 ``` ## ⚠️ NOT a plain deploy -- migrate first Deploying alone would present **empty** dirs to jellyfin/restic (data still sits in `/persist`). Before/with the deploy, on the guest: ``` systemctl stop jellyfin restic-backups-local restic-backups-rascal restic-backups-benland mkdir -p /storage/beefcake-cold/var/lib /storage/beefcake-cold/var/cache rsync -aHAX --info=progress2 /persist/var/lib/jellyfin/ /storage/beefcake-cold/var/lib/jellyfin/ rsync -aHAX --info=progress2 /persist/var/cache/restic-backups-local/ /storage/beefcake-cold/var/cache/restic-backups-local/ # ...rascal, benland likewise ``` then deploy, confirm the binds are live (`findmnt /var/lib/jellyfin`), verify jellyfin + a restic run, and only then remove the old copies from `/persist` to actually reclaim the SSD. Restic caches are rebuildable, so if you would rather not rsync 15G over virtiofs you can simply let restic repopulate them (first run after is slower). ## Deferred deliberately - `/root/.cache` (5.1G) nests inside the wholesale-persisted `/root`. - `/var/log` (1.7G) -- capping journald `SystemMaxUse` is the better lever. ## Verified Config evaluates; fstab carries the ordering option on all four binds; tmpfiles creates the backing dirs. **Not deployed** -- given I caused the outage earlier today, I did not want to ship an impermanence change to this host unvalidated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015dwmWkDcFN7CnZ5TmLK9nY

lytedev added 1 commit

2026-07-21 12:31:19 -05:00

feat(beefcake): move cold persist state off the SSD onto zstorage

/ check-format (push) Successful in 11s

Details

/ build (push) Successful in 6m12s

Details

6cf6c06094

battleship rpool (222G SSD, backing the guest zvols) hit 100% and paused the
guest on 2026-07-21. Audit of the guest 30.6G /persist showed the pressure is
NOT data worth keeping on SSD:

  var/cache/restic-backups-*  15.0G  systemd CacheDirectory, rebuildable
  root/.cache                  5.1G  cache
  var/lib/jellyfin             3.8G  regenerable metadata/artwork
  var/log                      1.7G  journal + caddy
  forgejo (repos+db+indexers)  1.3G  <- the actually-hot set

Move the restic caches and jellyfin metadata (~19G) to /storage, which is
virtiofs onto zstorage (2.4T free). They still survive reboots -- the original
comment persisted the restic caches deliberately so post-reboot runs do not
re-fetch indexes from the remote sftp repos, and that property is preserved.

forgejo deliberately STAYS on the SSD: virtiofs is ~430x slower for git (see
forgejo.nix) and PR #756 fixed 2-7s repo pages precisely by getting git off it.
Moving /persist wholesale to zstorage would have re-broken that while leaving
the caches on SSD -- exactly backwards.

Implemented as explicit fileSystems binds with depends = [ "/storage" ] rather
than a second environment.persistence root: impermanence emits its bind units
with DefaultDependencies=false and NO ordering against /storage, so they could
bind before the virtiofs mount exists and services would silently write to the
EPHEMERAL root and lose data at the next reboot. Verified the generated fstab
carries x-systemd.requires-mounts-for=/storage on all four binds.

Deferred deliberately: /root/.cache (5.1G) nests inside the wholesale-persisted
/root, and /var/log (1.7G) is better handled by capping journald SystemMaxUse.

NOT A PLAIN DEPLOY -- the data must be migrated first, see the PR body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015dwmWkDcFN7CnZ5TmLK9nY

lytedev merged commit d7e15b21c8 into main

2026-07-21 13:13:36 -05:00

lytedev deleted branch cold-persist

2026-07-21 13:13:37 -05:00