lytedev/nix
Watch
1
0
Fork
You've already forked nix
 2
Code Pull requests 10 Activity Actions

feat(deck-pose): gesture recognition and Unix socket publisher (phase 2) #907

Closed
lytedev wants to merge 3 commits from deck-pose-events into deck-pose
pull from: deck-pose-events
merge into: lytedev:deck-pose
Conversation 1 Commits 3 Files changed 12 +1717 -49
lytedev commented 2026-08-03 09:55:32 -05:00
Owner
Copy link

Phase 2 of deck-pose: the event/gesture layer and the publishing API.

Stacked on #906 — base is deck-pose, not main, because this builds directly on the phase-1 daemon and the events.rs seam it defined. Review/merge #906 first.

Nothing is deployed and nothing is enabled. Publishing is off unless explicitly asked for.

Recognisers (recognise.rs)

jump, squat, punch, swipe, dwell, plus player_joined/player_left. Three properties matter more than the individual thresholds:

  • Body-relative thresholds, scaled by shoulder-to-hip distance, so a gesture works at any distance from the camera rather than exactly one.
  • Every gesture has a refractory period. Without one a single physical motion spanning several frames fires an event on each of them and a game sees one punch as six.
  • Low-confidence keypoints are treated as absent, not as positions. MoveNet places uncertain joints somewhere plausible-looking but frequently wrong; believing them produces phantom gestures — the worst failure for an input device, because the player did nothing and the game reacted.

Velocities are per-second from frame timestamps, not per-frame deltas, so no threshold silently changes meaning when the frame rate does.

Swipe carries a unit direction vector, speed, and the wrist position at recognition time — a slice game has to hit-test the path, and by the time the event arrives the hand has moved on.

One deliberate deviation from the spec, flagged

Dwell carries a position rather than the region: u32 the phase-1 sketch imagined. Regions are a game's notion of its own UI; the daemon cannot know them without being configured with each game's layout, so hit-testing belongs on the game's side of the socket. Say the word if you want the region form back and I'll add a configurable region table.

Publisher (publish.rs)

Newline-delimited JSON over a Unix socket. Connect, read lines, parse each as one object — no handshake, no framing beyond the newline, so anything that can read lines can be a client.

The rule the module is built around: a slow or dead client must never slow the daemon down. The publish path does no I/O. It serialises once, fans the same Arc<str> to each client's writer thread through a bounded queue with try_send, and moves on. A full queue drops and counts the message; a client behind for a sustained run is disconnected rather than accumulating a backlog of stale poses it could not use anyway — the same newest-wins reasoning as the capture stage.

Both requested streams are there: a continuous per-frame pose message (all 17 keypoints per player, not just wrists — a few KB at six players, and a new game then needs no daemon change to use hip position) and discrete event messages. Events are stamped with their frame's capture instant, so events and poses share one clock and can be correlated.

Publishing is opt-in. A pose stream is a camera feed by another name and should not appear on a socket because someone ran the binary. --socket, [publish] socket, or lyte.deck-pose.socket in the NixOS module.

Measured on dragon, live camera + live client

  • 60.1fps, P50 11.7ms / P95 13.2ms — unchanged from phase 1 within run-to-run noise
  • recognition + publish: 0.02ms/frame, now its own PUB field in the readout
  • 602 messages in 10s (60/s), zero drops
  • a real dwell event fired from a hand held still:
    {"t":"event","seq":280,"ts_ms":8740.1,"player":1,"kind":"dwell","hand":"right","x":0.52,"y":0.79,"held_ms":715.2}

Checks

  • nix build .#deck-pose succeeds
  • 39 tests pass (was 21), all sandbox-safe — the socket tests use Unix sockets under a temp dir, no camera, no network, no tzdata
  • includes a test that publishes 2000 frames at a client that never reads and asserts it does not block
  • clippy, cargo fmt, nix fmt clean

Untested

  • More than one concurrent client. The fan-out is written for many and unit-tested for the never-reads case, but no two real clients have ever been attached.
  • No consumer with an opinion about the protocol. So far only socat has read it; the slice game is the real test.
  • Gesture thresholds are unvalidated by feel. dwell fired correctly on real motion; jump/squat/punch/swipe have unit tests over synthetic bodies but nobody has stood in front of the camera and tried to trigger them deliberately. Expect tuning.
  • Nothing has run on the steamdeck.
Phase 2 of deck-pose: the event/gesture layer and the publishing API. **Stacked on #906** — base is `deck-pose`, not `main`, because this builds directly on the phase-1 daemon and the events.rs seam it defined. Review/merge #906 first. Nothing is deployed and nothing is enabled. Publishing is off unless explicitly asked for. ## Recognisers (`recognise.rs`) `jump`, `squat`, `punch`, `swipe`, `dwell`, plus `player_joined`/`player_left`. Three properties matter more than the individual thresholds: - **Body-relative thresholds**, scaled by shoulder-to-hip distance, so a gesture works at any distance from the camera rather than exactly one. - **Every gesture has a refractory period.** Without one a single physical motion spanning several frames fires an event on each of them and a game sees one punch as six. - **Low-confidence keypoints are treated as absent, not as positions.** MoveNet places uncertain joints somewhere plausible-looking but frequently wrong; believing them produces phantom gestures — the worst failure for an input device, because the player did nothing and the game reacted. Velocities are per-second from frame timestamps, not per-frame deltas, so no threshold silently changes meaning when the frame rate does. `Swipe` carries a unit direction vector, speed, **and** the wrist position at recognition time — a slice game has to hit-test the path, and by the time the event arrives the hand has moved on. ## One deliberate deviation from the spec, flagged `Dwell` carries a **position** rather than the `region: u32` the phase-1 sketch imagined. Regions are a game's notion of its own UI; the daemon cannot know them without being configured with each game's layout, so hit-testing belongs on the game's side of the socket. Say the word if you want the region form back and I'll add a configurable region table. ## Publisher (`publish.rs`) Newline-delimited JSON over a Unix socket. Connect, read lines, parse each as one object — no handshake, no framing beyond the newline, so anything that can read lines can be a client. **The rule the module is built around: a slow or dead client must never slow the daemon down.** The publish path does no I/O. It serialises once, fans the same `Arc<str>` to each client's writer thread through a bounded queue with `try_send`, and moves on. A full queue drops and counts the message; a client behind for a sustained run is disconnected rather than accumulating a backlog of stale poses it could not use anyway — the same newest-wins reasoning as the capture stage. Both requested streams are there: a **continuous per-frame pose message** (all 17 keypoints per player, not just wrists — a few KB at six players, and a new game then needs no daemon change to use hip position) and **discrete event messages**. Events are stamped with their frame's capture instant, so events and poses share one clock and can be correlated. Publishing is **opt-in**. A pose stream is a camera feed by another name and should not appear on a socket because someone ran the binary. `--socket`, `[publish] socket`, or `lyte.deck-pose.socket` in the NixOS module. ## Measured on dragon, live camera + live client - **60.1fps, P50 11.7ms / P95 13.2ms** — unchanged from phase 1 within run-to-run noise - **recognition + publish: 0.02ms/frame**, now its own `PUB` field in the readout - 602 messages in 10s (60/s), zero drops - a real `dwell` event fired from a hand held still: `{"t":"event","seq":280,"ts_ms":8740.1,"player":1,"kind":"dwell","hand":"right","x":0.52,"y":0.79,"held_ms":715.2}` ## Checks - `nix build .#deck-pose` succeeds - **39 tests** pass (was 21), all sandbox-safe — the socket tests use Unix sockets under a temp dir, no camera, no network, no tzdata - includes a test that publishes 2000 frames at a client that never reads and asserts it does not block - clippy, `cargo fmt`, `nix fmt` clean ## Untested - **More than one concurrent client.** The fan-out is written for many and unit-tested for the never-reads case, but no two real clients have ever been attached. - **No consumer with an opinion about the protocol.** So far only `socat` has read it; the slice game is the real test. - **Gesture thresholds are unvalidated by feel.** `dwell` fired correctly on real motion; jump/squat/punch/swipe have unit tests over synthetic bodies but nobody has stood in front of the camera and tried to trigger them deliberately. Expect tuning. - Nothing has run on the steamdeck.
feat(deck-pose): gesture recognition and a Unix socket publisher
All checks were successful
/ check-format (push) Successful in 11s
Details
/ build (push) Successful in 6m18s
Details
df546a5734 
Phase 2: the seam sketched in events.rs is now implemented. The daemon
recognises gestures and publishes both a continuous per-frame pose stream and
discrete events as newline-delimited JSON, so games can be separate processes
that link no ML runtime.

Recognisers (recognise.rs) — jump, squat, punch, swipe, dwell, plus
player joined/left. Three properties matter more than the individual
thresholds:

* Thresholds are BODY-RELATIVE, scaled by shoulder-to-hip distance, so a
  gesture works at any distance from the camera rather than at exactly one.
* Every gesture has a REFRACTORY PERIOD. Without one a single physical motion
  spanning several frames fires an event on each of them and a game sees one
  punch as six.
* Low-confidence keypoints are treated as ABSENT, not as positions. MoveNet
  places uncertain joints somewhere plausible-looking but frequently wrong,
  and believing them produces phantom gestures — the worst failure mode for an
  input device, because the player did nothing and the game reacted.

Velocities are per-second, derived from frame timestamps rather than per-frame
deltas, so no threshold silently changes meaning when the frame rate does.

Swipe carries a unit direction vector, speed AND the wrist position at
recognition time: a slice game has to hit-test the path, and by the time the
event arrives the hand has moved on.

Dwell carries a position rather than the region id the phase-1 sketch
imagined. Regions are a game`s notion of its own UI; the daemon cannot know
them without being configured with a game`s layout, so hit-testing belongs on
the game`s side of the socket.

Publisher (publish.rs) — the rule that shapes it is that a slow or dead client
must NEVER slow the daemon down. The publish path does no I/O: it serialises
once, fans the same Arc out to each client`s writer thread through a bounded
queue with try_send, and moves on. A full queue drops the message and counts
it; a client behind for a sustained run is disconnected rather than
accumulating a backlog of stale poses it could not use anyway. Same newest-wins
reasoning as the capture stage. Tested by publishing 2000 frames at a client
that never reads.

Publishing is OFF unless asked for. A pose stream is a camera feed by another
name and should not appear on a socket because someone ran the binary.

Measured on dragon with a live client: 60.1fps, P50 11.7ms / P95 13.2ms,
recognition + publish 0.02ms/frame, 602 messages in 10s, zero drops. A real
dwell event fired from a hand held still.
fix(deck-pose): refuse to steal a socket another daemon is publishing on
All checks were successful
/ check-format (push) Successful in 10s
Details
/ build (push) Successful in 6m13s
Details
416567481f 
The reported symptom was a client counter reading zero while a client was
demonstrably receiving 60 messages a second. The counter was not the bug — it
was accurately reporting that THAT daemon had no clients.

`bind` unlinked whatever was at the socket path before binding. Unlinking a
live Unix socket does not disturb the process listening on it: its listener
keeps working on the now-unnamed inode, so the first daemon stays up, never
accepts another client, and reports zero forever, while a second daemon
started on the same path quietly serves everyone. Two daemons then disagree
about reality and the only visible symptom is the count. Reproduced exactly in
a test before changing anything: two publishers on one path gave first=0,
second=1, with the client receiving from the second.

Ownership is now an flock on a `<socket>.lock` sidecar. Deliberately not a
connect-probe of the socket: probing means connecting, which registers a
phantom client on the daemon being probed and reintroduces a smaller version
of the same lying counter — the first attempt at this fix did exactly that and
the test caught it. The kernel drops an flock when the holder dies by any
means, including SIGKILL, so an unclean shutdown still leaves the path usable.

Three tests: a second bind on a live socket fails and the incumbent keeps
serving; a stale socket file left by an unclean shutdown is still cleared; and
the client count reads 1 across sustained traffic rather than only at the
moment of connection.
feat(deck-pose): report published messages beside the client count
All checks were successful
/ check-format (push) Successful in 11s
Details
/ build (push) Successful in 6m37s
Details
2c27221192 
The client-count telemetry became the subject of a disagreement that could not
be settled after the fact: one observer saw `clients=0` while another saw a
client receiving 60 messages a second, and the log carried nothing to
reconcile them. Both observations were about different moments, but nothing on
the line said so.

The periodic line now reports `published`, the number of messages accepted by
at least one client`s queue. Read together the two numbers are no longer
ambiguous: zero clients beside a RISING published total is a genuine
contradiction worth investigating, and zero clients beside a flat total is
simply nobody listening — which is what a daemon logs for every cycle before a
game connects, and is almost certainly what was seen.

Also adds a test that checks the underlying invariant DIRECTLY instead of
arguing it from the code. "publish_frame early-returns at zero clients, so no
bytes can flow" is only as strong as the claim that every delivery path is
gated, which was the thing in doubt. The test publishes continuously with a
client attached and asserts, on every iteration where a message was actually
delivered, that the count was not zero at that instant — plus that nothing is
delivered at all before a client attaches.
lytedev commented 2026-08-03 11:29:53 -05:00
Author
Owner
Copy link

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against main. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against `main`. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.
lytedev closed this pull request 2026-08-03 11:29:58 -05:00
All checks were successful
/ check-format (push) Successful in 11s
Details
/ build (push) Successful in 6m37s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is wrong

  
enhancement
New feature

  
question
Awaiting information

  
scary
Potentially problematic change

  
upstream
Awaiting upstream changes

No labels
bug
enhancement
question
scary
upstream
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lytedev/nix!907
No description provided.