feat(deck-pose): markerless pose-input daemon (phase 1 spike)

lytedev commented

2026-08-02 23:42:34 -05:00

Owner

Phase 1 of a markerless full-body pose-input system for the steamdeck host — camera-only body tracking as a game input platform, modelled on the Nex Playground.

This is the latency spike only. No gesture layer, no socket API, no games; those are phase 2 and are only worth building if these numbers hold. Full plan and open questions: issues/open/deck-pose.md. Runbook, budget and rationale: lib/doc/deck-pose.md.

Nothing is deployed and nothing is enabled. The steamdeck host sets lyte.deck-pose.enable = false.

What is here

v4l2 capture → MJPEG/YUYV/GREY decode → MoveNet MultiPose Lightning → IoU tracking → One Euro smoothing → debug overlay with a live per-stage latency readout. Crane-built, packaged, NixOS module, docs, issue.

Two findings that contradicted the plan

TFLite is unreachable from this repo. nixpkgs.tensorflow-lite is marked broken, and every Rust TFLite binding either builds TensorFlow with bazel or downloads a prebuilt runtime from a build script — neither survives the nix sandbox. Took the documented fallback: ONNX Runtime via ort (load-dynamic, dlopening nixpkgs onnxruntime 1.26.0), with the same Apache-2.0 MoveNet weights as an ONNX re-export pinned by commit and hash.

int8 is ~12x SLOWER than float32, measured, not assumed:

Variant	192x192	256x256
float32	5.4ms	8.4ms
int8	62.6ms	100.4ms

ORT's CPU kernels leave the dynamic-quantised QDQ nodes unfused and fall back to slow integer convolutions. Default is float32; int8 stays packaged so the comparison can be repeated on the Deck.

Measured

On dragon (not the target), OBSBOT Meet 2 at 1280x720 MJPEG 60fps, 256x256 model input, headless:

60fps sustained, 0 frames dropped — the pipeline keeps up with the camera entirely
in-process latency P50 10.4ms / P95 10.7ms (budget: 80ms end-to-end)
breakdown: decode 2.2 / preprocess 0.5 / inference 7.7 / tracking <0.1

That figure is dequeue→present-returned. It excludes sensor exposure before dequeue and present→photons, both real and both invisible from inside the process — the doc says so explicitly rather than letting it read as glass-to-glass.

Model output layout ((y, x, score) x17 then the bbox) was validated empirically against a real person image, not taken on faith.

Design points worth reviewing

Capture drops stale frames rather than queueing them — a one-slot mailbox new frames overwrite. A FIFO between a 60fps camera and a slower consumer makes the acted-on pose steadily older without bound. A high DROPPED count is the mechanism working.
Player identity is tracked, not read off the model. MoveNet's six slots are not identities. Association is by bbox IoU with the assignment solved exactly (exhaustive search with pruning over ≤6x6) — greedy matching fails precisely when two players cross. IDs are never recycled.
One Euro filter per keypoint per player, adapting cutoff to speed, so jitter dies at rest without paying lag in motion.

Checks

nix build .#deck-pose — builds; wrapped binary runs against a real camera with the model and ORT baked in
19 unit tests pass (pure arithmetic — no camera, no model, no network, no tzdata, so they survive the crane sandbox)
cargo clippy --all-targets clean, cargo fmt and nix fmt clean
nixosConfigurations.steamdeck evaluates both with the module off (as committed) and forced on

Needs Daniel

hardware: nothing has run on the Deck; Van Gogh is 4-core Zen 2 at handheld power
a purchase decision: the camera is unchosen (wide FOV, 60fps, MJPEG, ideally global shutter — several candidates are mono, which is handled but untested for accuracy)
hands: real glass-to-glass needs a physical measurement, and keypoint accuracy for a whole body at living-room distance is unverified

Phase 1 of a markerless full-body pose-input system for the `steamdeck` host — camera-only body tracking as a game input platform, modelled on the Nex Playground. This is the **latency spike only**. No gesture layer, no socket API, no games; those are phase 2 and are only worth building if these numbers hold. Full plan and open questions: `issues/open/deck-pose.md`. Runbook, budget and rationale: `lib/doc/deck-pose.md`. **Nothing is deployed and nothing is enabled.** The steamdeck host sets `lyte.deck-pose.enable = false`. ## What is here v4l2 capture → MJPEG/YUYV/GREY decode → MoveNet MultiPose Lightning → IoU tracking → One Euro smoothing → debug overlay with a live per-stage latency readout. Crane-built, packaged, NixOS module, docs, issue. ## Two findings that contradicted the plan **TFLite is unreachable from this repo.** `nixpkgs.tensorflow-lite` is marked broken, and every Rust TFLite binding either builds TensorFlow with bazel or downloads a prebuilt runtime from a build script — neither survives the nix sandbox. Took the documented fallback: ONNX Runtime via `ort` (`load-dynamic`, dlopening nixpkgs `onnxruntime` 1.26.0), with the same Apache-2.0 MoveNet weights as an ONNX re-export pinned by commit and hash. **int8 is ~12x SLOWER than float32**, measured, not assumed: | Variant | 192x192 | 256x256 | | --- | --- | --- | | float32 | 5.4ms | 8.4ms | | int8 | 62.6ms | 100.4ms | ORT's CPU kernels leave the dynamic-quantised QDQ nodes unfused and fall back to slow integer convolutions. Default is float32; int8 stays packaged so the comparison can be repeated on the Deck. ## Measured On **dragon** (not the target), OBSBOT Meet 2 at 1280x720 MJPEG 60fps, 256x256 model input, headless: - **60fps sustained, 0 frames dropped** — the pipeline keeps up with the camera entirely - **in-process latency P50 10.4ms / P95 10.7ms** (budget: 80ms end-to-end) - breakdown: decode 2.2 / preprocess 0.5 / inference 7.7 / tracking <0.1 That figure is dequeue→present-returned. It excludes sensor exposure before dequeue and present→photons, both real and both invisible from inside the process — the doc says so explicitly rather than letting it read as glass-to-glass. Model output layout ((y, x, score) x17 then the bbox) was validated empirically against a real person image, not taken on faith. ## Design points worth reviewing - **Capture drops stale frames rather than queueing them** — a one-slot mailbox new frames overwrite. A FIFO between a 60fps camera and a slower consumer makes the acted-on pose steadily older without bound. A high `DROPPED` count is the mechanism working. - **Player identity is tracked, not read off the model.** MoveNet's six slots are not identities. Association is by bbox IoU with the assignment solved exactly (exhaustive search with pruning over ≤6x6) — greedy matching fails precisely when two players cross. IDs are never recycled. - **One Euro filter per keypoint per player**, adapting cutoff to speed, so jitter dies at rest without paying lag in motion. ## Checks - `nix build .#deck-pose` — builds; wrapped binary runs against a real camera with the model and ORT baked in - 19 unit tests pass (pure arithmetic — no camera, no model, no network, no tzdata, so they survive the crane sandbox) - `cargo clippy --all-targets` clean, `cargo fmt` and `nix fmt` clean - `nixosConfigurations.steamdeck` evaluates both with the module off (as committed) and forced on ## Needs Daniel - **hardware**: nothing has run on the Deck; Van Gogh is 4-core Zen 2 at handheld power - **a purchase decision**: the camera is unchosen (wide FOV, 60fps, MJPEG, ideally global shutter — several candidates are mono, which is handled but untested for accuracy) - **hands**: real glass-to-glass needs a physical measurement, and keypoint accuracy for a whole body at living-room distance is unverified

lytedev added 3 commits

2026-08-02 23:42:34 -05:00

feat(deck-pose): pose-input daemon spike — capture, infer, track, smooth 7971cecf4e

Phase 1 of a markerless full-body pose-input system for the steamdeck host
(camera-only body tracking as a game input platform, no controllers). This
change is the latency spike only: no gesture recognition, no publishing API,
no games. Those are phase 2, and are only worth building if the numbers here
hold — the point of the spike is to find out.

Pipeline: v4l2 capture -> MJPEG/YUYV/GREY decode -> MoveNet MultiPose
Lightning -> IoU tracking -> One Euro smoothing -> overlay with a live
per-stage latency readout.

Three decisions are load-bearing and were forced by measurement rather than
plan:

* ONNX Runtime, not TFLite+XNNPACK. `nixpkgs.tensorflow-lite` is marked
  broken, and every Rust TFLite binding either builds TensorFlow with bazel or
  downloads a prebuilt runtime from a build script — none of which survives
  the nix sandbox. `onnxruntime` is packaged and substitutes, so the model is
  consumed as an ONNX re-export of the same Apache-2.0 MoveNet weights.

* float32, not int8. Under ONNX Runtime`s CPU provider the int8 export of this
  model measures ~12x SLOWER than float32 (100ms vs 8.4ms per 256x256 pass) —
  its QDQ nodes fall back to unfused integer convolutions. Quantisation only
  pays when the runtime has kernels for it.

* Capture drops stale frames rather than queueing them. A FIFO handoff between
  a 60fps camera and a slower consumer makes the acted-on pose steadily older
  without bound; a one-slot mailbox that new frames overwrite keeps latency
  flat and makes the loss visible as a counter.

Measured on dragon (not the target hardware) against a real UVC camera at
1280x720: 60fps sustained, in-process latency P50 10.4ms / P95 10.7ms, zero
frames dropped. Budget is 80ms end-to-end.

feat(deck-pose): NixOS module, wired into steamdeck but disabled c70ef6ed12

The module installs the daemon and grants camera access; it does not start
anything. `service.enable` is a second, separate opt-in that runs it headless
at login — off until there is an event API worth having up before a game does.

`users` is empty by default rather than defaulting to the primary user:
handing a process the camera is a decision, not a convenience.

The steamdeck host sets `enable = false`. Nothing here has run on that
hardware, and the camera has not been chosen — `/dev/video0` is a guess on a
machine with no built-in camera, so the device must be set before enabling.

The module records, in comments, the two things phase 1 deliberately does not
solve: the overlay cannot appear over a gamescope session (the eventual answer
is that the daemon has no window and games read its socket), and continuous
CPU inference on a handheld has an unbudgeted power cost.

docs(deck-pose): runbook, latency budget and model rationale

/ check-format (push) Successful in 10s

Details

/ build (push) Successful in 6m8s

Details

cc2845127b

Records the measurements and the two findings that contradict the plan, so
neither has to be rediscovered:

* TFLite is unreachable here (nixpkgs.tensorflow-lite is broken; every Rust
  binding needs bazel or a downloading build script), hence ONNX Runtime.
* int8 is ~12x SLOWER than float32 under ORT`s CPU provider, with the numbers
  behind that claim.

Also states plainly what the reported latency does NOT include — sensor
exposure before dequeue, and present-to-photons after — so the in-process
figure is not mistaken for glass-to-glass.

The issue file carries the phased plan and the open questions, each marked with
what it needs: hardware, a purchase decision, or physical testing.

lytedev added 1 commit

2026-08-03 09:43:52 -05:00

fix(deck-pose): scale the One Euro defaults to normalised coordinates

/ check-format (push) Successful in 11s

Details

/ build (push) Successful in 6m9s

Details

ffe5a6435b

The defaults were the Casiez et al. paper values (min_cutoff 1.0, beta 0.007),
which are wrong here — and wrong in a way that looks right, since anyone who
recognises them will assume they are correct.

The adaptive term is `cutoff = min_cutoff + beta * |velocity|`, so beta`s
correct magnitude depends on the units velocity is measured in. The paper tunes
for PIXEL-valued mouse input moving at hundreds of units/second. These
keypoints are normalised to 0..1, so a fast hand crossing half the frame in
150ms moves at ~3.3 units/second. With beta = 0.007 the adaptive term is worth
~0.02Hz against a 1.0Hz baseline: the filter degenerates into a fixed 1Hz
low-pass (tau 159ms, alpha 0.095 at 60fps) and the skeleton visibly lerps
toward the body rather than tracking it. Observed live on dragon.

min_cutoff 1.5 / beta 3.0 restores the intended adaptive behaviour at this
scale, confirmed on real motion. Both are now real CLI flags, along with
derivative_cutoff: these are tuned by feel, and requiring a TOML file to try a
value is the wrong ergonomics for that.

smooth.rs gains a regression test that simulates the swipe and asserts the
shipped defaults both track it closely and beat the paper values, so the
degenerate case cannot come back silently. The smoothing tests now use
SmoothingConfig::default() rather than a private copy of the numbers, so they
cannot drift away from what actually ships.

lytedev referenced this pull request

2026-08-03 09:55:32 -05:00

feat(deck-pose): gesture recognition and Unix socket publisher (phase 2) #907

lytedev referenced this pull request

2026-08-03 10:06:10 -05:00

feat(deck-slice): Fruit-Ninja-style slice game driven by deck-pose (phase 3) #908

lytedev referenced this pull request

2026-08-03 10:46:02 -05:00

feat(deck-slice): spatial framing, self-view and edge cues #909

lytedev referenced this pull request

2026-08-03 11:29:34 -05:00

WIP: markerless full-body pose input (deck-pose daemon + deck-slice game) #915

lytedev commented

2026-08-03 11:29:45 -05:00

Author

Owner

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against main. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against `main`. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

lytedev closed this pull request

2026-08-03 11:29:49 -05:00

/ check-format (push) Successful in 11s

Required

Details

/ build (push) Successful in 6m9s

Required

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Rows
Columns

feat(deck-pose): markerless pose-input daemon (phase 1 spike) #906

What is here

Two findings that contradicted the plan

Measured

Design points worth reviewing

Checks

Needs Daniel

Pull request closed