feat(deploy): take a host lease so deploys cannot clobber in-flight validation #777

Merged

lytedev merged 1 commit from deploy-host-lock into main

2026-07-17 10:11:57 -05:00

lytedev commented

2026-07-17 04:12:00 -05:00

Owner

Implements invariant #2 from #775: never clobber validation someone else has in progress. The downgrade guard already covers invariant #1 (never move a host backward unknowingly); this is the other half.

A host runs one generation at a time, so deploying to a host somebody is still verifying silently destroys their work — and the host gives no hint verification was happening.

Key design decision: the lease is held past the deploy

The work being protected is the verification that happens after deploy-rs exits, not the switch itself. An auto-released lock would leave open exactly the window it exists to close. So:

released explicitly — deploy --release-lock --targets ".#beefcake"
TTL-expires (default 60m) so a crashed agent can't wedge the fleet
lives in /run (tmpfs) → a rebooted host starts unlocked, which is correct: nothing can be mid-validation across a reboot
held on both success and failure — on success you're verifying; on failure the host may be half-changed. Neither is a good time for someone else to deploy over it.

Holder identity is per-workspace, not per-user

$USER@$(hostname) would make every agent on this box look like daniel@dragon and never block each other — defeating the point. Each agent works in its own jj workspace, so the workspace dir name is a natural per-agent identity that's also stable across the several deploy runs one agent makes while iterating (its own lease re-acquires rather than blocking).

deploy-guard: 'beefcake' lease taken as 'daniel@dragon:deploy-host-lock' (ttl 60m)

Blocked output

============================== DEPLOY BLOCKED ==============================
'beefcake' is leased by another deployer — deploying now could destroy work
they are still validating on the host.

  holder: daniel@dragon:blue-green
  held:   12m ago

Wait for them, or if you know the lease is abandoned:
  deploy --force-lock ...           steal it
  deploy --release-lock --targets ".#beefcake"   release it
It also expires on its own, and is cleared by a host reboot.
===========================================================================

Notes

Honors --hostname when a single host is targeted, so the lease lands on the box actually being deployed — deploying beefcake over the LAN (--hostname 192.168.0.9) is the documented norm and must work with the VPN down.
Fails open when a host is unreachable, matching the existing guard: this is a courtesy guard against clobbering a teammate, not a security control, and must never block a legitimate deploy on its own.
Remote helpers are piped in as real bash (ssh … bash -s) — root's login shell on these hosts is fish, which would mis-parse an inline bash one-liner.
Escape hatches: --force-lock, --release-lock, --no-lock / LYTE_DEPLOY_NO_LOCK, LYTE_DEPLOY_LOCK_TTL, LYTE_DEPLOY_HOLDER.

A real wedge found while testing

If an acquire died between the mkdir and writing meta (ssh dropping mid-call), the lockdir existed with no valid meta → since=0, ttl=0. The stale check requires ttl > 0, so the lease could never expire and reported a ~56-year age — precisely the failure the TTL exists to prevent. Missing/garbled meta now falls back to the lockdir's mtime for age and the caller's ttl for expiry, so a half-written lease ages out and self-heals.

Verification

Extracted the shipped scripts from the built store artifact and drove the state machine:

case	result
a takes free lease	`ACQUIRED`
b blocked by a	`HELD\|agent-a\|0`
a re-deploys (own lease)	`REACQUIRED`
b `--force-lock` steals	`FORCED\|agent-a\|0`
a can't release b's	`NOT_YOURS\|agent-b`
b releases own	`RELEASED`
release when free	`NOT_HELD`
ttl expiry	`BROKE_STALE\|agent-a\|2`
crash-mid-acquire, fresh	`HELD\|unknown (incomplete lease)\|0`
crash-mid-acquire, aged	`BROKE_STALE` → self-heals

Arg paths (--hostname override, --release-lock not deploying, --no-lock) checked via LYTE_DEPLOY_GUARD_DRYRUN=1. Builds clean through writeShellApplication's shellcheck.

Not yet exercised against a real host — the remote half is proven by running the shipped scripts locally against a temp lockdir, not over ssh to beefcake. Worth one live acquire/release on a non-critical host before merging.

Implements invariant #2 from #775: **never clobber validation someone else has in progress.** The downgrade guard already covers invariant #1 (never move a host backward unknowingly); this is the other half. A host runs **one generation at a time**, so deploying to a host somebody is still verifying silently destroys their work — and the host gives no hint verification was happening. ## Key design decision: the lease is held *past* the deploy The work being protected is the **verification that happens after `deploy-rs` exits**, not the switch itself. An auto-released lock would leave open exactly the window it exists to close. So: - released **explicitly** — `deploy --release-lock --targets ".#beefcake"` - **TTL-expires** (default 60m) so a crashed agent can't wedge the fleet - lives in `/run` (tmpfs) → a rebooted host starts unlocked, which is correct: nothing can be mid-validation across a reboot - held on **both success and failure** — on success you're verifying; on failure the host may be half-changed. Neither is a good time for someone else to deploy over it. ## Holder identity is per-*workspace*, not per-user `$USER@$(hostname)` would make every agent on this box look like `daniel@dragon` and **never block each other** — defeating the point. Each agent works in its own jj workspace, so the workspace dir name is a natural per-agent identity that's also stable across the several `deploy` runs one agent makes while iterating (its own lease **re-acquires** rather than blocking). ``` deploy-guard: 'beefcake' lease taken as 'daniel@dragon:deploy-host-lock' (ttl 60m) ``` ## Blocked output ``` ============================== DEPLOY BLOCKED ============================== 'beefcake' is leased by another deployer — deploying now could destroy work they are still validating on the host. holder: daniel@dragon:blue-green held: 12m ago Wait for them, or if you know the lease is abandoned: deploy --force-lock ... steal it deploy --release-lock --targets ".#beefcake" release it It also expires on its own, and is cleared by a host reboot. =========================================================================== ``` ## Notes - **Honors `--hostname`** when a single host is targeted, so the lease lands on the box actually being deployed — deploying beefcake over the LAN (`--hostname 192.168.0.9`) is the documented norm and must work with the VPN down. - **Fails open** when a host is unreachable, matching the existing guard: this is a courtesy guard against clobbering a teammate, not a security control, and must never block a legitimate deploy on its own. - Remote helpers are piped in as real bash (`ssh … bash -s`) — root's login shell on these hosts is **fish**, which would mis-parse an inline bash one-liner. - Escape hatches: `--force-lock`, `--release-lock`, `--no-lock` / `LYTE_DEPLOY_NO_LOCK`, `LYTE_DEPLOY_LOCK_TTL`, `LYTE_DEPLOY_HOLDER`. ## A real wedge found while testing If an acquire died between the `mkdir` and writing `meta` (ssh dropping mid-call), the lockdir existed with no valid meta → `since=0`, `ttl=0`. The stale check requires `ttl > 0`, so the lease could **never expire** and reported a ~56-year age — precisely the failure the TTL exists to prevent. Missing/garbled meta now falls back to the lockdir's **mtime** for age and the caller's ttl for expiry, so a half-written lease ages out and self-heals. ## Verification Extracted the shipped scripts from the **built store artifact** and drove the state machine: | case | result | |---|---| | a takes free lease | `ACQUIRED` | | b blocked by a | `HELD\|agent-a\|0` | | a re-deploys (own lease) | `REACQUIRED` | | b `--force-lock` steals | `FORCED\|agent-a\|0` | | a can't release b's | `NOT_YOURS\|agent-b` | | b releases own | `RELEASED` | | release when free | `NOT_HELD` | | ttl expiry | `BROKE_STALE\|agent-a\|2` | | crash-mid-acquire, fresh | `HELD\|unknown (incomplete lease)\|0` | | crash-mid-acquire, aged | `BROKE_STALE` → self-heals | Arg paths (`--hostname` override, `--release-lock` not deploying, `--no-lock`) checked via `LYTE_DEPLOY_GUARD_DRYRUN=1`. Builds clean through `writeShellApplication`'s shellcheck. **Not yet exercised against a real host** — the remote half is proven by running the shipped scripts locally against a temp lockdir, not over ssh to beefcake. Worth one live acquire/release on a non-critical host before merging.

lytedev added 1 commit

2026-07-17 04:12:01 -05:00

feat(deploy): take a host lease so deploys cannot clobber in-flight validation

/ check-format (push) Successful in 11s

Details

/ build (push) Successful in 6m8s

Details

bd00508f07

A host runs exactly ONE generation at a time, so deploying to a host somebody
else is still verifying silently destroys their work — and the host gives no
hint that verification was in progress. The downgrade guard already covers
"never move a host backward unknowingly"; this covers the other half: "never
clobber validation someone else has in progress".

The lease is deliberately held PAST the deploy. The work being protected is the
verification that happens AFTER deploy-rs exits, not the switch itself, so an
auto-released lock would leave exactly the window it exists to close. It is
therefore released explicitly (--release-lock) and TTL-expires (default 60m) so
a crashed agent cannot wedge the fleet. It lives in /run (tmpfs), so a rebooted
host starts unlocked — correct, since nothing can be mid-validation across a
reboot. Held on both success and failure: on success you are verifying, on
failure the host may be half-changed, and in neither case should someone else
deploy over it.

Holder identity is $USER@$host:<workspace-dir>, not $USER@$host. Every agent on
this box runs as daniel@dragon, so that alone would make them all look like the
same holder and never block each other. Each agent works in its own jj
workspace, so the workspace name is a natural per-agent identity that is also
stable across the several deploy runs one agent makes while iterating (its own
lease re-acquires rather than blocking).

Honors an explicit --hostname override when a single host is targeted, so the
lease lands on the box actually being deployed. This matters because deploying
beefcake over the LAN (--hostname 192.168.0.9) is the documented norm, and the
lease must work there even when the VPN is down.

Fails OPEN whenever a host is unreachable, matching the existing guard: this is
a courtesy guard against clobbering a teammate, not a security control, and must
never block a legitimate deploy on its own.

Remote helpers are piped in as real bash (ssh ... bash -s) because root login
shell on these hosts is fish, which would mis-parse an inline bash one-liner.

Escape hatches: --force-lock steals, --release-lock hands back, --no-lock /
LYTE_DEPLOY_NO_LOCK opts out, LYTE_DEPLOY_LOCK_TTL tunes expiry,
LYTE_DEPLOY_HOLDER overrides identity.

Testing found and fixed a real wedge: if an acquire died between the mkdir and
writing meta (ssh dropping mid-call), the lockdir existed with no valid meta, so
since=0 and ttl=0 — the stale check requires ttl>0, so the lease could NEVER
expire and reported a ~56-year age. That is precisely the failure the TTL exists
to prevent. Missing/garbled meta now falls back to the lockdirs mtime for age
and the callers ttl for expiry, so a half-written lease ages out and self-heals.

Verified by extracting the shipped scripts from the built store artifact and
driving the state machine: acquire, block, re-acquire own, force-steal,
release-not-yours, release-own, release-when-free, ttl expiry, and both the
fresh and aged crash-mid-acquire cases. Arg paths (--hostname override,
--release-lock not deploying, --no-lock) checked via LYTE_DEPLOY_GUARD_DRYRUN.

Docs for the invariant land in the linked docs PR (#775).

lytedev referenced this pull request

2026-07-17 10:08:39 -05:00

docs(agents): reconcile agent instructions; scope global vs repo #775

lytedev merged commit dc9ea8077e into main

2026-07-17 10:11:57 -05:00

lytedev deleted branch deploy-host-lock

2026-07-17 10:11:57 -05:00