Touch input: the same board, driven by a finger

[Deyvis17GY]

What I saw

The stage listens to MediaPipe and nothing else — there is no pointer,
touch, or mouse handler in the page. On a phone or a touchscreen the board
renders and then cannot be used at all: nothing grabs, nothing scrolls,
nothing taps.

Hand tracking works, and it is the point of the project. But it tires the
hands over a long session, and it needs a camera, light, and a secure
context. On a tablet in the hand, the finger is right there.

What I expected instead

A finger on an item behaves like a pinch on it: drag to move, quick tap to
open or close, two fingers to scale, a flick to throw.

How to reproduce

Open stage.html on any touchscreen — I used a phone over a Tailscale
HTTPS proxy, so the secure-context requirement was satisfied and the
camera was available — and try to move a card with a finger. Nothing
responds.

The quickest confirmation is in the source rather than on a device:
grepping stage.html for pointerdown, touchstart or mousedown
returns nothing. The only input path is MediaPipe.

Setup

Linux host serving the repo's server.py on 127.0.0.1:8794, reached from
an Android phone over a Tailscale HTTPS proxy. Chrome. barehands v1.0
(eb23bed).

The cause, and what I did about it

Every downstream behaviour already reads a cursor, never a hand:
hitTest, beginGrab, endGrab and the drag/scroll/stretch block all
take (i, cur) and never ask where cur came from. So touch does not need
a second implementation — it needs to fill the same object.

Two things were in the way:

1. The held-item drive lives inside the hand loop. The
   drag / scroll / stretch block sits inside the per-hand forEach, so a
   non-hand cursor can hold an item but never move it. Lifting it verbatim
   into driveHeld(i, cur) fixes that without changing a line of its
   behaviour.

2. Cursor bookkeeping is numeric. The loop drops cursors it stopped
   seeing with Object.keys(cursors).map(Number). A touch cursor keyed
   "p3" is NaN to that sweep, so the camera can never delete a finger.
   That is the whole trick, and it is why hands and fingers coexist with no
   arbitration between them.

With those two, the touch layer is small: pointerdown sets
cur.pinched = true and calls beginGrab on the hit; pointermove moves
the cursor, pushes a history sample and calls driveHeld; pointerup
calls dropCursor, which already ends the grab. Tap-to-open,
tap-to-close, the two-hand stretch (now two fingers) and the peak-velocity
fling all came along for free — endGrab never knew the difference.

Two details worth keeping if you build it:

• A parked finger still has to tick. The hold-still-to-rotate latch and
  the scroll-to-move promotion are time based, and endGrab reads the
  history to decide fling versus set-down. Without a per-frame top-up, a
  finger that travels fast, stops, and lifts gets read as a throw, because
  the newest samples in the history are the ones from the travel.
• touch-action: none on the body, or the browser pans the page out
  from under the drag.

I also added ?hands=0, which skips pickInitialCam() and the landmarker
entirely and guards the detection block with if (landmarker && ...). On a
phone that means no camera permission, no WASM download, and no per-frame
inference — the board just runs. ?touch=0 opts back out of touch.

Happy to paste the full diff, or to leave it here if you would rather write
it in your own style — it is your pipeline, and the shape above is the part
that matters.

[rathlavatharun117-a11y]

hey