feat(deck-slice): spatial framing, self-view and edge cues #909

Closed

lytedev wants to merge 2 commits from deck-slice-framing into deck-slice

pull from: deck-slice-framing

merge into: lytedev:deck-slice

lytedev:main

lytedev:propose/fallback-fail-fast

lytedev:propose/detect-agent-death

lytedev:propose/local-fallback

lytedev:propose/local-inference

lytedev:propose/bc250-image

lytedev:pinephone-niri

lytedev:propose/herdr-keybinds

lytedev:propose/spoof-reject-cleanups

lytedev:propose/finance-pipeline

lytedev:propose/jj-ripe

lytedev:propose/reminder-history

lytedev:steamframe-plan

lytedev:deck-pose-wip

lytedev:pinephone-buildable

lytedev:aerc-open-html

lytedev:local-auth-unification

lytedev:esphome-music-panel

lytedev:wifi-system-profile

lytedev:gauge-history

lytedev:validate-gate-fix2

lytedev:validate-slot-no-hugepages

lytedev:pinephone-reenable

lytedev:cyberdeck-research

lytedev:git-anon-ratelimit

lytedev:headscale-basedomain

lytedev:ssd-fast-tier

lytedev:hold-impermanence-flip

lytedev:zen-browser

lytedev:initrd-remote-unlock

lytedev:beefcake-rpool-backup

lytedev:sec-sso-plan

lytedev:sec-mail

lytedev:flab-lid-wake

lytedev:hearth-k8s

lytedev:zulip

lytedev:router-guest-vlan

lytedev:headscale-extra-records

lytedev:beefcake-advertise-lan

lytedev:push-qrvuuqmxtznv

lytedev:voxtype-per-host-model

lytedev:cleanup

lytedev:push-kanidm-opt-in

lytedev:push-voxtype-flake-fix

lytedev:push-uvqqknxvqlvy

lytedev:push-qrqsxkmlrevert

lytedev:fix-kanidm-uid-attr-map

lytedev:nix-on-droid

lytedev:self-hosted-dns

lytedev:fix-runner-tmpfs-revert

lytedev:beefcake-email

lytedev:beefcake-immich

lytedev:firefox-nightly-devedition

lytedev:fix-bun-baseline

lytedev:flab-stuff

lytedev:printer-sftp

lytedev:dragon-vr-monado

lytedev:clever-vr-game-clamshell-mode-detection

lytedev:wireguard-mesh

lytedev:rescue-beefcake-unpushed-nix

Conversation 1 Commits 2 Files changed 13 +525 -24

lytedev commented 2026-08-03 10:46:02 -05:00

Owner

Copy link

Addresses Daniel's feedback after the live test: "I need a sense of space, and I think the window versus camera's viewport makes the ratios feel confusing? and showing the skeleton at least a tad might help provide that."

Stacked on #908 (which is on #907, which is on #906). Base is deck-slice.

Priority note: the letterbox/viewport-ratio fix (part 1 below) is the approved, prioritised piece. The self-view and edge cues (parts 2 and 3) were small — a few hundred lines of pure drawing code with no coupling to the mapping — so they landed alongside rather than being deferred. Nothing was held up to include them. If you want part 1 mergeable on its own anyway, say so and I will split them; I have not rewritten the pushed commits to do that unasked.

1. Letterbox the camera aspect (the prioritised fix)

Pose coordinates are normalised against the camera frame. Mapping 0..1 onto a viewport of a different aspect scales x and y by different factors, so identical physical hand movement covered different screen distance horizontally vs vertically — exactly the "ratios feel confusing" symptom.

The camera aspect is now letterboxed inside the viewport, and every pose→screen mapping goes through that rect. The aspect is derived from the w/h carried on each pose message rather than hardcoded, since the daemon negotiates the capture format at runtime and does not always get what it asked for. A mid-stream resolution change recomputes the rect, as does a window resize.

The rect is made visible: everything outside it is dimmed and it carries a faint border, so "where the camera can see me" is a place rather than an inference. Targets spawn inside it, so nothing is thrown where a hand cannot reach.

Consequence worth flagging: the cut-speed threshold is now in play-rect widths per second, not pixels. A fixed pixel threshold silently demanded a faster physical swing on a larger window. The number changed meaning, so it may want retuning once someone plays it.

2. Self-view

Each player's skeleton drawn faintly behind the gameplay — low alpha, thin bones, deliberately subordinate to the targets. Blades say where your hands are but nothing about where you are; a player who has drifted sideways or stood too close otherwise cannot tell. Toggle with --no-self-view or the S key, with a hint in the corner since there is no menu.

3. Edge cues

When a hand leaves the frame or drops below the confidence threshold, the edge it was last seen near glows. A hand merely approaching an edge glows it weakly, as a warning. Without this, "my hand left the frame", "the model lost my hand" and "the game has hung" all look identical — a blade that stopped moving. Naming the direction is what turns the loss into something the player can act on.

Mirroring — already correct, verified by hand

MIRROR_X was already true and the behaviour was right: raise your right hand and the blade on your right moves. Daniel confirmed this directly by playing it. Unchanged in this PR, but now pinned by a test so it cannot regress silently.

Tests

The mapping arithmetic is extracted into framing.gd as pure static functions, covered by tests/framing_test.gd, which the nix build runs — 13 checks including the uniform-scaling property this change exists to establish:

  ok   fills the width of a taller window
  ok   letterboxes the height
  ok   axes scale by the camera's aspect, not the window's
  ok   a hand on the camera's left appears on the right
  ok   zero camera size is empty
  ...
framing: all checks passed

Sandbox-safe by construction — no camera, socket, network or clock. I verified Godot propagates the script exit code by deliberately failing a check (exit 1) rather than assuming it, so this can actually fail a build.

The letterbox is now verified at runtime, not just in unit tests

A gap I found while firming this up: every end-to-end run so far used a 16:9 viewport, where letterboxing is a no-op — so the runtime path was effectively untested even though the arithmetic was covered.

The resolved play rect is now logged on every change (startup, resize, camera renegotiation), which both closes that gap and makes future "the ratios feel wrong" reports diagnosable. A headless run turns out to use a square 1280x1280 viewport, which exercises it properly:

deck-slice: camera 1280x720 in window 1280x1280 -> play rect 0,280 1280x720
deck-slice: connected to /tmp/claude-1000/fake-pose5.sock
deck-slice: P1 sliced target, score -8

Inset 280px top and bottom, as it should be, with spawning-inside-the-rect and slicing working against it.

Also fixed in passing

The scene gained an Entities node so runtime targets and blades draw between the self-view and the HUD. They were previously appended after the HUD and drew over the score.

Untested

• Not run against a real camera. /dev/video0 was held by the daemon you left running for Daniel (PID 3335421), so I did not touch it; this round was validated against the synthetic pose stream.
• Nobody has looked at it. The mapping is tested and the rect is now confirmed correct at runtime, but whether the vignette, skeleton alpha and edge glow actually deliver a sense of space is a judgement only playing it can make. Expect the alpha values in particular to need taste applied — they are my guess, not a tuned result.

Addresses Daniel's feedback after the live test: *"I need a sense of space, and I think the window versus camera's viewport makes the ratios feel confusing? and showing the skeleton at least a tad might help provide that."* **Stacked on #908** (which is on #907, which is on #906). Base is `deck-slice`. **Priority note:** the letterbox/viewport-ratio fix (part 1 below) is the approved, prioritised piece. The self-view and edge cues (parts 2 and 3) were small — a few hundred lines of pure drawing code with no coupling to the mapping — so they landed alongside rather than being deferred. Nothing was held up to include them. If you want part 1 mergeable on its own anyway, say so and I will split them; I have not rewritten the pushed commits to do that unasked. ## 1. Letterbox the camera aspect (the prioritised fix) Pose coordinates are normalised against the camera frame. Mapping 0..1 onto a viewport of a *different* aspect scales x and y by different factors, so identical physical hand movement covered different screen distance horizontally vs vertically — exactly the "ratios feel confusing" symptom. The camera aspect is now letterboxed inside the viewport, and **every** pose→screen mapping goes through that rect. The aspect is derived from the `w`/`h` carried on each pose message rather than hardcoded, since the daemon negotiates the capture format at runtime and does not always get what it asked for. A mid-stream resolution change recomputes the rect, as does a window resize. The rect is made visible: everything outside it is dimmed and it carries a faint border, so "where the camera can see me" is a place rather than an inference. Targets spawn inside it, so nothing is thrown where a hand cannot reach. **Consequence worth flagging:** the cut-speed threshold is now in play-rect widths per second, not pixels. A fixed pixel threshold silently demanded a faster physical swing on a larger window. The number changed meaning, so it may want retuning once someone plays it. ## 2. Self-view Each player's skeleton drawn faintly behind the gameplay — low alpha, thin bones, deliberately subordinate to the targets. Blades say where your hands are but nothing about where *you* are; a player who has drifted sideways or stood too close otherwise cannot tell. Toggle with `--no-self-view` or the `S` key, with a hint in the corner since there is no menu. ## 3. Edge cues When a hand leaves the frame or drops below the confidence threshold, the edge it was last seen near glows. A hand merely *approaching* an edge glows it weakly, as a warning. Without this, "my hand left the frame", "the model lost my hand" and "the game has hung" all look identical — a blade that stopped moving. Naming the direction is what turns the loss into something the player can act on. ## Mirroring — already correct, verified by hand `MIRROR_X` was already `true` and the behaviour was right: raise your right hand and the blade on your right moves. **Daniel confirmed this directly by playing it.** Unchanged in this PR, but now pinned by a test so it cannot regress silently. ## Tests The mapping arithmetic is extracted into `framing.gd` as pure static functions, covered by `tests/framing_test.gd`, which **the nix build runs** — 13 checks including the uniform-scaling property this change exists to establish: ``` ok fills the width of a taller window ok letterboxes the height ok axes scale by the camera's aspect, not the window's ok a hand on the camera's left appears on the right ok zero camera size is empty ... framing: all checks passed ``` Sandbox-safe by construction — no camera, socket, network or clock. I verified Godot propagates the script exit code by deliberately failing a check (exit 1) rather than assuming it, so this can actually fail a build. ## The letterbox is now verified at runtime, not just in unit tests A gap I found while firming this up: every end-to-end run so far used a 16:9 viewport, where letterboxing is a **no-op** — so the runtime path was effectively untested even though the arithmetic was covered. The resolved play rect is now logged on every change (startup, resize, camera renegotiation), which both closes that gap and makes future "the ratios feel wrong" reports diagnosable. A headless run turns out to use a *square* 1280x1280 viewport, which exercises it properly: ``` deck-slice: camera 1280x720 in window 1280x1280 -> play rect 0,280 1280x720 deck-slice: connected to /tmp/claude-1000/fake-pose5.sock deck-slice: P1 sliced target, score -8 ``` Inset 280px top and bottom, as it should be, with spawning-inside-the-rect and slicing working against it. ## Also fixed in passing The scene gained an `Entities` node so runtime targets and blades draw between the self-view and the HUD. They were previously appended after the HUD and drew over the score. ## Untested - **Not run against a real camera.** `/dev/video0` was held by the daemon you left running for Daniel (PID 3335421), so I did not touch it; this round was validated against the synthetic pose stream. - **Nobody has looked at it.** The mapping is tested and the rect is now confirmed correct at runtime, but whether the vignette, skeleton alpha and edge glow actually *deliver* a sense of space is a judgement only playing it can make. Expect the alpha values in particular to need taste applied — they are my guess, not a tuned result.

lytedev added 1 commit 2026-08-03 10:46:02 -05:00

feat(deck-slice): spatial framing, self-view and edge cues ... 20ef278c77

Daniel: "I need a sense of space, and I think the window versus camera`s
viewport makes the ratios feel confusing? and showing the skeleton at least a
tad might help provide that." Three changes, addressing the two separate
problems in that.

LETTERBOX THE CAMERA ASPECT. Pose coordinates are normalised against the
camera frame; mapping 0..1 onto a viewport of a different aspect scales x and
y by different factors, so identical physical hand movement covered different
screen distance horizontally and vertically. The camera aspect is now
letterboxed inside the viewport and EVERY pose->screen mapping goes through
that rect. The aspect comes from the `w`/`h` on each pose message rather than a
constant, because the daemon negotiates the capture format at runtime and does
not always get what it asked for.

The rect is visible — everything outside it dimmed, a faint border on it — so
"where the camera can see me" is a place rather than an inference. Targets
spawn inside it, so nothing is thrown where a hand cannot reach.

Consequence worth noting: the cut-speed threshold is now expressed in
play-rect widths per second rather than pixels. A fixed pixel threshold
silently demanded a faster physical swing on a larger window.

SELF-VIEW. Each player`s skeleton drawn faintly behind the gameplay — low
alpha, thin bones, deliberately subordinate to the targets. Blades say where
your hands are but nothing about where YOU are, and a player who has drifted
sideways or stood too close otherwise cannot tell. Toggleable with
`--no-self-view` or `S`.

EDGE GLOW. When a hand leaves frame or drops below the confidence threshold,
the edge it was last seen near glows; a hand merely approaching an edge glows
it weakly. Without this, "my hand left the frame", "the model lost my hand" and
"the game has hung" all look identical — a blade that stopped moving.

MIRRORING was already correct (MIRROR_X) and is unchanged; there is now a test
pinning it so it cannot regress silently.

The mapping arithmetic is extracted into framing.gd as pure static functions
and covered by tests/framing_test.gd, which the nix build runs — including the
uniform-scaling property this change exists to establish. Godot propagates the
script`s exit code, verified by deliberately failing a check.

Scene gained an Entities node so runtime targets and blades draw between the
self-view and the HUD; previously they were appended after the HUD and drew
over the score.

lytedev added 1 commit 2026-08-03 10:48:13 -05:00

feat(deck-slice): log the resolved play rect ... f182a46dd3

"The ratios feel wrong" can only be diagnosed against the rect actually in
use, and nothing was reporting it. Logged on every change — startup, window
resize, camera renegotiation — which is rare enough to be worth a line.

This also closed a real gap in how the letterbox was verified. The unit tests
cover the arithmetic, but every end-to-end run so far used a 16:9 viewport
where letterboxing is a no-op, so the runtime path was effectively untested.
With the rect logged, a headless run turns out to have a SQUARE 1280x1280
viewport, which exercises it properly: camera 1280x720 gives a play rect at
0,280 sized 1280x720 — inset 280px top and bottom, as it should be — and
gameplay (spawning inside the rect, slicing, scoring) works against it.

lytedev force-pushed deck-slice-framing from f182a46dd3 to 59851455b5 2026-08-03 10:57:44 -05:00 Compare

lytedev referenced this pull request from a commit 2026-08-03 10:57:44 -05:00

fix(deck-slice): make the cut threshold body-relative and forgiving

lytedev referenced this pull request 2026-08-03 10:58:30 -05:00

fix(deck-slice): make the cut threshold body-relative and forgiving #910

lytedev referenced this pull request from a commit 2026-08-03 11:01:58 -05:00

fix(deck-slice): make the cut threshold body-relative and forgiving

lytedev force-pushed deck-slice-framing from 59851455b5 to 23ab92880f 2026-08-03 11:01:59 -05:00 Compare

lytedev referenced this pull request 2026-08-03 11:29:34 -05:00

WIP: markerless full-body pose input (deck-pose daemon + deck-slice game) #915

lytedev commented 2026-08-03 11:30:12 -05:00

Author

Owner

Copy link

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against main. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

Superseded by #915, which consolidates the whole markerless-pose-input program into a single WIP branch against `main`. Every commit from this PR is preserved there — nothing was squashed, so the root-cause writeups in the commit messages are intact. Closing here; review happens on #915.

lytedev closed this pull request 2026-08-03 11:30:16 -05:00

lytedev referenced this pull request from a commit 2026-08-04 15:35:30 -05:00

fix(deck-slice): make the cut threshold body-relative and forgiving

All checks were successful

Hide all checks

/ check-format (push) Successful in 10s

Details

/ build (push) Successful in 6m31s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is wrong

  

enhancement

New feature

  

question

Awaiting information

  

scary

Potentially problematic change

  

upstream

Awaiting upstream changes

No labels

bug

enhancement

question

scary

upstream

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lytedev/nix!909

No description provided.