feat(deck-pose,deck-slice): draw the camera feed as the game background #913

Closed

lytedev wants to merge 1 commit from deck-video into deck-slice-speed

lytedev commented

2026-08-03 11:19:02 -05:00

Owner

Daniel's request: "I think we should draw the actual camera feed as the background so users can feel how the tracking relates to their bodies."

Stacked on #911.

Your design — agreed, with one rejection

I agree with almost all of it and implemented it as specified: forward the original bytes, separate socket, opt-in and off by default, rate-limited independently, drop-never-queue, drawn in the play rect mirrored and dimmed.

I did not implement daemon-side downscale, because it contradicts "do not re-encode". Any change to the pixels — including making them smaller — means decode, scale and JPEG encode on the latency-critical path, to produce a worse image than the one already in hand. There is no cheap downscale of a JPEG.

Resolution is instead chosen where it is free: at the camera, via the existing --width/--height, which shrinks the capture, the forward and the consumer's decode all at once. --video-fps (default 30) caps the rate as you asked. If you still want a daemon-side resize I will add it, but it should be a knowing trade of daemon latency for consumer convenience, not a default.

Measured, as requested

Daemon cost, isolated on dragon (720p MJPEG, video capped at 30fps):

condition	inference
video off	8.0 ms	baseline
`--publish-video`, no consumer	8.1 ms	the no-consumer path is a true no-op
actively forwarding	8.7 ms	+0.7 ms — the JPEG memcpy

The pose path is unaffected: with the game attached and decoding, the daemon holds 60.1 fps, P95 15.4 ms end-to-end, unchanged from baseline and far inside the 80 ms budget.

GDScript decode cost: 4-7 ms per 720p frame, the dominant cost of the whole feature, and it lands on the game rather than the input path. The game held 120 fps throughout.

One honest caution. Those are desktop numbers. In one earlier run under heavy contention I saw daemon inference stretch to 24-29 ms with the game decoding concurrently — not reproducible in the controlled runs, but a handheld doing inference and a game and a 30fps JPEG decode on four cores is a different proposition. --video-fps and a smaller capture are the levers, which is precisely why both are configurable.

Does it subsume the vignette?

Partly, and I changed it accordingly rather than stacking redundant cues. With the feed up the image itself shows where the camera can see, so the explicit border drops to a hairline. The vignette stays only to darken what the camera cannot see, which is still doing real work. Without video, both revert to their previous strength.

Implementation notes

Wire format: 12-byte header (magic DPV1, length, width, height) then the JPEG. Length-prefixed because JPEG has no line structure; self-describing so a consumer reconnecting mid-stream can tell it is synchronised rather than guessing, and reconnect rather than scan if not.
Only the newest complete frame in the buffer is decoded, the rest discarded — decoding a backlog spends the frame budget catching up to an image nobody will see. Same newest-wins discipline as capture.
The ImageTexture is reused rather than recreated per frame; allocating a GPU texture 30 times a second would have made this look expensive when the decode is the real cost.
Entirely optional and independent: if the socket never appears the game plays exactly as before.

Checks

nix build green for both packages. 50 daemon tests pass, including: header integrity and byte-for-byte payload equality, non-MJPEG formats skipped rather than converted, the rate limit holding against a burst, nothing sent with no consumer, and 500 frames offered at a consumer that never reads without blocking. Module evaluates with video on and emits the right config.

Untested

Nobody has seen it. It runs end to end against the real camera and the decode numbers are real, but whether the feed at 0.45 dim actually gives the sense of body-to-tracking relationship Daniel wanted is a judgement only playing it can make. The dim level is a guess.
Never run on the steamdeck, where the CPU contention caveat above actually matters.
Only one video consumer has ever been attached.

Daniel's request: *"I think we should draw the actual camera feed as the background so users can feel how the tracking relates to their bodies."* **Stacked on #911.** ## Your design — agreed, with one rejection I agree with almost all of it and implemented it as specified: forward the original bytes, separate socket, opt-in and off by default, rate-limited independently, drop-never-queue, drawn in the play rect mirrored and dimmed. **I did not implement daemon-side downscale, because it contradicts "do not re-encode".** Any change to the pixels — including making them *smaller* — means decode, scale and JPEG **encode** on the latency-critical path, to produce a worse image than the one already in hand. There is no cheap downscale of a JPEG. Resolution is instead chosen where it *is* free: at the camera, via the existing `--width`/`--height`, which shrinks the capture, the forward and the consumer's decode all at once. `--video-fps` (default 30) caps the rate as you asked. If you still want a daemon-side resize I will add it, but it should be a knowing trade of daemon latency for consumer convenience, not a default. ## Measured, as requested Daemon cost, isolated on dragon (720p MJPEG, video capped at 30fps): | condition | inference | | | --- | --- | --- | | video off | 8.0 ms | baseline | | `--publish-video`, no consumer | 8.1 ms | the no-consumer path is a true no-op | | actively forwarding | 8.7 ms | **+0.7 ms** — the JPEG memcpy | **The pose path is unaffected**: with the game attached and decoding, the daemon holds **60.1 fps, P95 15.4 ms** end-to-end, unchanged from baseline and far inside the 80 ms budget. **GDScript decode cost: 4-7 ms per 720p frame**, the dominant cost of the whole feature, and it lands on the game rather than the input path. The game held 120 fps throughout. **One honest caution.** Those are desktop numbers. In one earlier run under heavy contention I saw daemon inference stretch to 24-29 ms with the game decoding concurrently — not reproducible in the controlled runs, but a handheld doing inference *and* a game *and* a 30fps JPEG decode on four cores is a different proposition. `--video-fps` and a smaller capture are the levers, which is precisely why both are configurable. ## Does it subsume the vignette? **Partly, and I changed it accordingly rather than stacking redundant cues.** With the feed up the image itself shows where the camera can see, so the explicit border drops to a hairline. The vignette stays only to darken what the camera *cannot* see, which is still doing real work. Without video, both revert to their previous strength. ## Implementation notes - Wire format: 12-byte header (magic `DPV1`, length, width, height) then the JPEG. Length-prefixed because JPEG has no line structure; self-describing so a consumer reconnecting mid-stream can tell it is synchronised rather than guessing, and reconnect rather than scan if not. - Only the **newest** complete frame in the buffer is decoded, the rest discarded — decoding a backlog spends the frame budget catching up to an image nobody will see. Same newest-wins discipline as capture. - The `ImageTexture` is reused rather than recreated per frame; allocating a GPU texture 30 times a second would have made this look expensive when the decode is the real cost. - Entirely optional and independent: if the socket never appears the game plays exactly as before. ## Checks `nix build` green for both packages. **50 daemon tests** pass, including: header integrity and byte-for-byte payload equality, non-MJPEG formats skipped rather than converted, the rate limit holding against a burst, nothing sent with no consumer, and 500 frames offered at a consumer that never reads without blocking. Module evaluates with video on and emits the right config. ## Untested - **Nobody has seen it.** It runs end to end against the real camera and the decode numbers are real, but whether the feed at 0.45 dim actually gives the sense of body-to-tracking relationship Daniel wanted is a judgement only playing it can make. The dim level is a guess. - **Never run on the steamdeck**, where the CPU contention caveat above actually matters. - Only one video consumer has ever been attached.

lytedev added 1 commit

2026-08-03 11:19:02 -05:00

feat(deck-pose): forward camera video on a second socket

/ check-format (push) Successful in 10s

Details

/ build (push) Successful in 6m22s

Details

082adf20d4

Daniel: "I think we should draw the actual camera feed as the background so
users can feel how the tracking relates to their bodies."

The game cannot open the camera — V4L2 gives exclusive access to one opener and
the daemon holds it, the same constraint that made every stray daemon block the
next one during development. So video comes through the daemon or not at all.

FORWARDED, NEVER RE-ENCODED. The camera already delivers MJPEG and the daemon
already holds that buffer before decoding for inference, so forwarding costs a
memcpy. Measured on dragon: inference 8.0ms with video off, 8.1ms with
--publish-video and no consumer attached (a true no-op), 8.7ms while actively
forwarding. The pose path is unaffected — 60.1fps, P95 15.4ms end to end with
a game attached, against an 80ms budget.

NO DAEMON-SIDE DOWNSCALE, which is a deliberate rejection of the obvious
feature rather than an omission. Any change to the pixels — including making
them smaller — means decode, scale and JPEG ENCODE on the latency-critical
path, to produce a worse image than the one already in hand. Resolution is
chosen where it is free, at the camera via --width/--height, which shrinks the
capture, the forward and the consumer`s decode together. --video-fps caps the
rate independently of inference. Non-MJPEG formats are skipped, not converted.

Separate socket, not the NDJSON pose stream: video is binary and wants
length-prefixed framing, and base64 in a text protocol would be waste on the
input-critical channel. 12-byte header (magic, length, width, height) so a
consumer reconnecting mid-stream can tell it is synchronised rather than
guessing.

Opt-in and off by default, wired through the NixOS module as a decision
separate from pose publishing. The pose stream is coordinates; this is video of
someone`s living room.

Same drop-never-queue discipline as capture and the pose publisher, with a
one-frame queue: a stalled video consumer loses frames rather than adding a
millisecond to the input path. Tested by offering 500 frames at a consumer that
never reads.

lytedev referenced this pull request

2026-08-03 11:29:34 -05:00

WIP: markerless full-body pose input (deck-pose daemon + deck-slice game) #915

lytedev commented

2026-08-03 11:30:37 -05:00

Author

Owner

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against main. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against `main`. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

lytedev closed this pull request

2026-08-03 11:30:41 -05:00