fix(beefcake/forgejo): move git subprocess HOME off virtiofs to SSD #756

Merged

lytedev merged 1 commit from forgejo-git-home into main

2026-07-13 10:43:45 -05:00

lytedev commented

2026-07-13 09:46:20 -05:00

Owner

Problem

Every repo-scoped forgejo page has a fixed ~1-2s latency floor independent of how much git work it does: repo home ~2.5s, a raw single-file view ~1-2.7s, commits list 5-7s. Non-repo pages (/explore, static assets) serve in 2-35ms. Standalone git ops on the repo run in ~5ms, ARC hit 99.8%, last-commit cache warm — so it is not git, disk, or cache.

Root cause

forgejo's git HOME defaults to APP_DATA_PATH/home = /storage/forgejo/data/home, on the /storage virtiofs (HDD) share. Every git child forgejo spawns opens $HOME/.gitconfig and probes $HOME/.config/git/config at startup, and each such syscall costs 20-100ms over virtiofs vs <1ms on SSD. A single repo page opens the cat-file --batch/--batch-check readers (each preceded by git rev-parse in ensureValidGitRepository) plus hash-object/rev-list — ~6 git spawns even for a trivial raw-file request — so the per-request cost was ~6x the virtiofs config-read tax: a fixed floor unrelated to actual git work.

Diagnosis (live LTS 15.0.3 guest, pprof + execution trace)

Request goroutine blocked in os/exec on git children; go tool trace attributed the syscall-blocking to git.(*Command).Run, with ensureValidGitRepository's rev-parse alone = 2.12s of 2.84s.
Go scheduler latency ~40ms and SQLite ~10ms — negligible, refuting the DB single-writer contention hypothesis.
git rev-parse in the repo: 0.008-0.013s with HOME on tmpfs/SSD vs 0.135-0.835s with HOME on virtiofs; strace showed individual .gitconfig access/open/read syscalls at 20-98ms each on virtiofs.
Live A/B pointing git HOME at SSD cut commits 5-7s -> 2.5-3.5s and raw 1-2.7s -> 0.6-1.3s.

Fix

Set [git] HOME_PATH to /var/lib/forgejo-git-home on the guest SSD root, mirroring forgejoWorkTmp (tmpfiles-created, forgejo-owned, in ReadWritePaths, ephemeral — forgejo rewrites the managed .gitconfig on every start, so no persistence entry needed).

Residual / follow-up

After this fix the remaining per-request cost is the raw fork/exec of the many short-lived git subprocesses per repo page. Fully eliminating the floor would require reducing spawn count or faster storage for stateDir/WORK_DIR — tracked separately.

Deploy note

Config-only + a new ephemeral tmpfiles dir; safe as a live switch (no cross-release/systemd change). Not yet deployed.

🤖 Generated with Claude Code

https://claude.ai/code/session_011m7LpmJkaSpr9NiKvaatNv

## Problem Every repo-scoped forgejo page has a fixed ~1-2s latency floor independent of how much git work it does: repo home ~2.5s, a raw single-file view ~1-2.7s, commits list 5-7s. Non-repo pages (`/explore`, static assets) serve in 2-35ms. Standalone git ops on the repo run in ~5ms, ARC hit 99.8%, last-commit cache warm — so it is not git, disk, or cache. ## Root cause forgejo's git `HOME` defaults to `APP_DATA_PATH/home` = `/storage/forgejo/data/home`, on the `/storage` **virtiofs (HDD)** share. Every git child forgejo spawns opens `$HOME/.gitconfig` and probes `$HOME/.config/git/config` at startup, and each such syscall costs **20-100ms over virtiofs vs <1ms on SSD**. A single repo page opens the cat-file `--batch`/`--batch-check` readers (each preceded by `git rev-parse` in `ensureValidGitRepository`) plus `hash-object`/`rev-list` — **~6 git spawns even for a trivial raw-file request** — so the per-request cost was ~6x the virtiofs config-read tax: a fixed floor unrelated to actual git work. ## Diagnosis (live LTS 15.0.3 guest, pprof + execution trace) - Request goroutine blocked in `os/exec` on git children; `go tool trace` attributed the syscall-blocking to `git.(*Command).Run`, with `ensureValidGitRepository`'s `rev-parse` alone = **2.12s of 2.84s**. - Go scheduler latency ~40ms and SQLite ~10ms — negligible, **refuting the DB single-writer contention hypothesis**. - `git rev-parse` in the repo: **0.008-0.013s** with HOME on tmpfs/SSD vs **0.135-0.835s** with HOME on virtiofs; strace showed individual `.gitconfig` access/open/read syscalls at 20-98ms each on virtiofs. - Live A/B pointing git HOME at SSD cut commits 5-7s -> 2.5-3.5s and raw 1-2.7s -> 0.6-1.3s. ## Fix Set `[git] HOME_PATH` to `/var/lib/forgejo-git-home` on the guest SSD root, mirroring `forgejoWorkTmp` (tmpfiles-created, forgejo-owned, in `ReadWritePaths`, ephemeral — forgejo rewrites the managed `.gitconfig` on every start, so no persistence entry needed). ## Residual / follow-up After this fix the remaining per-request cost is the raw fork/exec of the many short-lived git subprocesses per repo page. Fully eliminating the floor would require reducing spawn count or faster storage for `stateDir`/`WORK_DIR` — tracked separately. ## Deploy note Config-only + a new ephemeral tmpfiles dir; safe as a live switch (no cross-release/systemd change). Not yet deployed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011m7LpmJkaSpr9NiKvaatNv

lytedev added 1 commit

2026-07-13 09:46:28 -05:00

fix(beefcake/forgejo): move git subprocess HOME off virtiofs to SSD

/ check-format (push) Successful in 17s

Details

/ build (push) Successful in 6m26s

Details

f107a254f5

Every repo-scoped forgejo page had a fixed ~1-2s latency floor independent
of how much git work it did: the repo home ~2.5s, a raw single-file view
~1-2.7s, the commits list 5-7s, while non-repo pages (/explore, static
assets) served in 2-35ms. Standalone git ops on the same repo ran in ~5ms,
ARC hit rate was 99.8%, and the last-commit cache was warm — so it was not
git, disk, or cache.

Root cause: forgejo's git HOME defaults to APP_DATA_PATH/home =
/storage/forgejo/data/home, which lives on the /storage virtiofs (HDD)
share. Every git child forgejo spawns opens $HOME/.gitconfig and probes
$HOME/.config/git/config at startup, and each of those syscalls costs
20-100ms over virtiofs vs <1ms on SSD. A single repo page opens the
cat-file --batch / --batch-check readers (each preceded by a `git rev-parse`
in ensureValidGitRepository) plus hash-object/rev-list, i.e. ~6 git spawns
even for a trivial raw-file request — so the per-request cost was ~6 x the
virtiofs config-read tax, a fixed floor unrelated to actual git work.

Diagnosis (pprof goroutine dumps + execution trace on the live LTS 15.0.3
guest): the request goroutine blocked in os/exec on git children;
go-tool-trace attributed the syscall-blocking to git command Run, with
ensureValidGitRepository's rev-parse alone accounting for 2.12s of 2.84s.
Go scheduler latency (~40ms) and SQLite (~10ms) were negligible, refuting
the DB-contention hypothesis. Measured git rev-parse in the repo: 0.008-0.013s
with HOME on tmpfs/SSD vs 0.135-0.835s with HOME on the virtiofs path; strace
showed individual .gitconfig access/open/read syscalls at 20-98ms each on
virtiofs. A live A/B pointing git HOME at SSD cut commits from 5-7s to
2.5-3.5s and raw from 1-2.7s to 0.6-1.3s.

Fix: set [git] HOME_PATH to /var/lib/forgejo-git-home on the guest SSD root,
mirroring forgejoWorkTmp (tmpfiles-created, forgejo-owned, in ReadWritePaths,
ephemeral — forgejo rewrites the managed .gitconfig on every start, so no
persistence entry is needed).

Residual: after this fix the remaining per-request cost is the raw fork/exec
of the many short-lived git subprocesses forgejo spawns per repo page; fully
eliminating the floor would require reducing spawn count or faster storage
for the stateDir/WORK_DIR, tracked separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011m7LpmJkaSpr9NiKvaatNv

lytedev merged commit 89770d9508 into main

2026-07-13 10:43:45 -05:00

lytedev deleted branch forgejo-git-home

2026-07-13 10:43:48 -05:00

lytedev referenced this pull request from a commit

2026-07-21 12:30:50 -05:00

feat(beefcake): move cold persist state off the SSD onto zstorage

lytedev referenced this pull request

2026-07-21 12:31:18 -05:00

feat(beefcake): move cold persist state off the SSD onto zstorage #801

lytedev referenced this pull request from a commit

2026-07-21 13:13:37 -05:00