WIP: markerless full-body pose input (deck-pose daemon + deck-slice game) #915
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "deck-pose-wip"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
WIP — not for merge. Consolidates the whole markerless-pose-input program into one branch so it can be reviewed and shipped as a chunk rather than as seven stacked PRs. Supersedes #906, #907, #908, #909, #910, #911 and #913, all now closed.
15 commits, deliberately not squashed — the messages carry the root-cause writeups (the unlink-a-live-socket mechanism, the dt-collapse analysis, the One Euro unit error) that a squash would destroy.
Nothing is deployed and nothing is enabled on any host.
What this is
Camera-only full-body tracking as a game input platform for the
steamdeck, modelled on the Nex Playground: a webcam, no controllers, up to six people. A daemon owns the camera and the model and publishes over Unix sockets; games are separate processes that link no ML runtime. Plus one game,deck-slice, that proves the seam works.packages/rust/deck-pose/— docslib/doc/deck-pose.mdpackages/games/deck-slice/— docslib/doc/deck-slice.mdissues/open/deck-pose.mdlib/modules/nixos/deck-pose.nix, wired intosteamdeckbut disabledThe stack, in order
StreamPeerUDS.Findings worth keeping
ONNX Runtime, not TFLite. TFLite was the intended runtime and is not reachable here:
nixpkgs.tensorflow-liteis marked broken, and every Rust binding either builds TensorFlow with bazel or downloads a prebuilt runtime in a build script — neither survives the nix sandbox. Consumed as an ONNX re-export of the same Apache-2.0 MoveNet weights, pinned by commit and hash. Ultralytics YOLO-pose was rejected on licensing (AGPL-3.0).int8 is ~12x SLOWER than float32, measured, not assumed — 100.4ms vs 8.4ms at 256x256. ORT's CPU kernels leave dynamic-quantisation QDQ nodes unfused. Quantisation only pays when the runtime has kernels for it.
The One Euro defaults were on the wrong scale. The paper's
beta = 0.007is tuned for pixel-valued mouse input; these keypoints are normalised 0..1, so the adaptive term contributed ~0.02Hz against a 1.0Hz baseline and the filter collapsed into a fixed 1Hz low-pass. The skeleton visibly lerped. Regression-tested.A daemon that unlinks a live socket lies about its client count. Unlinking does not disturb the process listening on it: the first daemon stays up, never accepts another client, and reports zero forever while a second serves everyone. Ownership is now an
flocksidecar, and the telemetry reportspublishedbesideclientsso the two can never again be reconciled only in hindsight.Hand speed was fabricated two different ways. (a) Measured against the game's receive time, where batched poses share a millisecond and dt collapses to zero — a gentle wave read as 4 body-heights/sec and sliced. (b) A teleported keypoint at an honest 16.7ms interval — MoveNet swapping wrists as arms cross, occlusion reacquisition, tracker identity swaps — computing ~94 body-heights/sec and slicing. Both fed
is_cutting(), so both caused phantom slices. Fixed by timing against the daemon's capture clock and rejecting physically impossible samples.Video is forwarded, never re-encoded. There is no cheap downscale of a JPEG: any pixel change means decode + scale + encode on the latency-critical path. Resolution is chosen at the camera, where it is free.
Measured (dragon, OBSBOT at 1280x720 MJPEG 60fps)
The latency figure is dequeue→present-returned. It excludes sensor exposure and present-to-photons, both real and neither visible from inside the process — so it is not glass-to-glass.
Verified
nix buildgreen for.#deck-poseand.#deck-slice. 50 daemon tests and 13 Godot framing checks, all sandbox-safe (no camera, network, tzdata). Daniel has play-tested the game live on a real camera and scored steadily.Untested / open
lib/doc/deck-slice.md, not solved.Camera lifecycle
Socket-activated, not always-on. systemd holds the listening sockets; the first connection starts the daemon; an idle timeout exits it when the last client leaves, releasing the camera. Always-on was the first design and was wrong: it would have held the camera open — LED on, watching the room — and run inference on a handheld's battery for the whole session, benefiting nobody while nobody plays. It also reintroduced by the back door exactly the exposure that made
--publish-videoopt-in.Verified on dragon: daemon opened the camera, exited 4s after the last client disconnected, and
fuser /dev/video0afterwards reported no holder.The two modes differ in who owns the socket path, which is modelled explicitly rather than inferred. Self-bound we clean up and guard with an flock; activated, systemd owns the path — unlinking it on idle exit would make activation work exactly once, with every later connection finding nothing listening.
Hotplug is what makes activation coherent rather than merely deferred: a daemon started by a connection still waits patiently when no camera is plugged in, and survives a mid-session unplug.
Two findings worth keeping
Probing beats reading the capability flag, and the flag genuinely cannot do the job.
VIDIOC_QUERYCAPreportscapabilities— the union across every node the device owns — alongside per-nodedevice_caps, and thev4lcrate exposes only the union. So/dev/video0(video capture) and/dev/video1(metadata capture) both claimVIDEO_CAPTUREand the flag cannot distinguish them. Negotiating a capture format can: the metadata node refuses. That is a test whose success implies the thing we actually care about, rather than one that correlates with it. This trap had bitten the project twice.Polling beats a udev monitor here, for the same reason. A
video4linuxadd event names a node but not which sibling it is — the metadata node fires an identical event — so the probe decides either way and udev could only ever be a hint to probe sooner. Polling on the existing reopen backoff does the same work with nolibudevin the closure and no extra descriptor to poll, which is a real benefit on a handheld. Removal is noticed immediately regardless, because the capture thread starts erroring on dequeue. If the 2s worst case ever proves too slow, udev drops in to trigger the probe early rather than replacing anything.Known upstream defects — filed, not fixed
Written up in
issues/open/deck-pose-phantom-gestures.md(included in this branch) rather than left in a PR description, since that is not somewhere a defect survives.The daemon emits phantom gestures on tracking discontinuities, today. It computes gesture speeds from raw keypoint deltas with no plausibility check, so a jumping keypoint produces a fictional speed that clears every threshold and publishes
swipe/punchevents to every consumer. Checking the code rather than trusting the original report turned up two corrections:jumpis exposed too (recognise.rs:270, threshold 1.2), andbody_scale— the denominator of all of them — is the vertical span alone floored at 0.05, so a lean or a turn foreshortens it toward zero. Conversely the daemon does not share the interval bug fixed game-side; it times off captureInstants with nanosecond resolution. Recorded so nobody re-fixes that.PlayerIddoes not keep its promise. It is documented as stable across occlusion and never recycled, and games are told to bind to it, but association is bbox IoU alone — which cannot separate two people crossing, since that is exactly when boxes overlap and only the poses differ. Adding keypoint similarity (OKS) to the cost function intrack.rsis the fix.The game-side guard in this branch protects the slice game's cut test only. It does nothing for the events the daemon publishes, and nothing for the next game.
The Deck has no built-in camera, so the normal case is a USB webcam plugged into an already-running system and unplugged again later. The daemon resolved one path at startup and exited on failure, which makes it unusable there: enabling it as a service on a Deck with nothing plugged in would crash-loop. Absence is now a STATE, not an error. The daemon starts without a camera, waits, picks one up when it appears, keeps running when it vanishes, and reacquires it — sockets bound and clients connected throughout. Reopen attempts back off to a 2s cap so an idle machine polls quietly instead of spinning. `--device auto` is the DEFAULT. /dev/video0 is not reliably the camera: the development OBSBOT presents /dev/video0 (video capture) and /dev/video1 (METADATA capture), and opening the wrong one fails with "querying current format" — a trap that has bitten this project twice. The capability flag cannot settle it, which is worth recording because it looks like it should. VIDIOC_QUERYCAP reports `capabilities` (the union across every node the DEVICE owns) and `device_caps` (this node only); the v4l crate exposes only the union, so both nodes claim VIDEO_CAPTURE. What distinguishes them is asking the node to do the job — query and set a capture format. The metadata node refuses. That is both the honest test and the one whose success implies the thing we actually care about. Identities come from /dev/v4l/by-id where available, since /dev/videoN renumbers across replug and across cameras; "the camera came back" should be a judgement about identity, not about a number that may now mean something else. Consumers are told explicitly rather than left to infer absence from silence: a `{"t":"camera","present":false,"device":...}` message on transitions, AND kept as sticky state replayed to every client the moment it connects. A game that starts during an outage would otherwise sit in a silence indistinguishable from a wedged daemon. POLLING, NOT UDEV, and the reasoning is in supervisor.rs. A video4linux add event names a node but not whether it is the capture node — the metadata sibling fires an identical event — so telling them apart means probing anyway. A udev event could only ever be a hint to run the probe sooner. Polling the probe on the existing backoff does the same work with no libudev in the closure and no extra descriptor, and removal is noticed immediately regardless because the capture thread starts erroring on dequeue. If a 2s worst case ever proves too slow, a udev monitor drops in to trigger attempt_open early rather than replacing any of this. Verified against the real camera on dragon: auto-select found usb-..._OBSBOT_Meet_2-video-index0 and ran at 60fps; starting with no camera kept the daemon alive with its socket bound and handed a late-joining client `present:false` as its first line; pointing explicitly at the metadata node reported absence instead of crashing.Daniel: "make sure the game can start the daemon and retries appropriately so we just launch it from the steam os menu". I evaluated your architecture rather than just building it, and agree with it. Hotplug is what unlocks it: the daemon no longer exits without a camera, so an always-up user service stops being a crash-loop and "launch from the menu" collapses into "launch the game". Service policy is now Restart=always rather than on-failure, which is only reasonable BECAUSE absence is no longer a failure. deck-slice-launch runs `systemctl --user start deck-pose.service` and then EXECS the game, so Steam`s process tree holds the game rather than a shell babysitting a child. Three things about it are deliberate: * systemctl rather than spawning a daemon ourselves — supervision, restart and logging already exist and a game that forks a daemon owns a lifecycle problem it should not. Starting an already-running unit is a no-op, and the flock makes even a real double-start safe. * Failing to start the service is NOT fatal. The game already retries the socket indefinitely and says on screen what it is waiting for, so a Deck without the unit still launches into something that explains itself. * The game is referenced by absolute store path, not via PATH. A gamescope session need not inherit a login shell`s PATH, and "works from a terminal, fails from the menu" is the exact trap the launcher exists to avoid. I had written the PATH version first. Game side handles the camera coming and going without a restart: the explicit `{"t":"camera"}` state clears the frozen video frame and stops drawing blades at positions nobody occupies, and the HUD distinguishes "no camera" from "no daemon" — different problems with different things for the player to do. Verified end to end: the game against a daemon with no camera connects, reports "camera absent" from the sticky protocol message rather than inferring it from silence, and keeps running. A .desktop entry is installed, which is what SteamOS`s Add a Non-Steam Game browser lists. Adding it to Steam stays manual — Steam owns that library and a nix build has no supported way in.Testing this against a real systemd user unit rather than only unit tests found three defects, two of which no unit test could have caught. CONFIG.TOML MUST NOT CARRY A PER-USER SOCKET PATH. It is one system-wide file shared by every user, so neither representation is correct: `%t` is systemd unit syntax and reaches the daemon as a literal directory, and an expanded /run/user/1000 is simply wrong for everyone else. Both options offered were traps. The config now expresses INTENT — publish, publish video — and the path comes from whoever places the socket: systemd binds %t/... and hands over the descriptor, or the daemon falls back to $XDG_RUNTIME_DIR. %t expands to exactly $XDG_RUNTIME_DIR, so the two agree BY CONSTRUCTION rather than by keeping two strings in step. A `%`-prefixed value is now recognised as unit-only syntax and never written to config. THE DAEMON IGNORED SYSTEMD`S SOCKET. Publishing required a config flag, so an activated daemon adopted nothing, served nothing, and then idled out while the triggering connection was still queued — so systemd re-activated it immediately. A hot restart loop, every 5 seconds, visible only against a real unit. Being handed a listening socket IS the request to publish on it. A DEPARTED CLIENT WAS NEVER REAPED WITHOUT TRAFFIC. Liveness was only noticed on write failure and pruning only happened during broadcast — but with no camera there are no writes and no broadcasts, so a client that had left still counted forever, the idle timeout never fired, and the daemon never released the camera. On a Deck, which has no built-in camera, that is the DEFAULT case. Writer threads now bound their wait and probe for EOF, and the count includes only live clients. Also: the module`s device option defaulted to /dev/video0, silently overriding the auto-selection the host config believed was in effect. Now `auto`. And a warning if a per-uid runtime path is configured: systemd.user units are installed for every user, so an absolute /run/user/<uid> path makes each user`s manager — root`s included — bind the same file. It looks correct until a second user logs in, which is exactly how this surfaced. VERIFIED end to end against a real systemd user socket unit on dragon: socket armed with the service inactive and no camera open; connecting started the service and the client`s FIRST line was {"t":"camera","seq":0,"present":false} — activation, hotplug absence and the protocol in one; 12s later the service was inactive again, having released the camera; the socket stayed armed and a second connection re-activated cleanly, proving the socket had not been unlinked.1bf8a5d2fdf5e3d66454View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.