FingerTrack

FingerTrack is a Python application that detects hand/finger states using a webcam and outputs them to configurable consumers.

It supports:

• Stable finger state detection with smoothing.
• Optional live OpenCV window display.
• Modular consumer system (stdout, HTTP, gRPC, etc.).
• Easy integration with other applications.

Features

• Detects which fingers are up per hand.
• Debouncing/smoothing over configurable history size.
• Output can be sent to multiple consumers:
  • Stdout
  • OpenCV window
  • HTTP API or custom consumers
• Optional continuous window refresh without affecting other consumers.
• Graceful shutdown on Ctrl+C or external SIGTERM.

Quickstart (Local)

git clone <repo-url>
cd fingertrack
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python app.py --preview-mode full

• --preview-mode enables the live OpenCV feed.
• The finger states are printed to stdout by default.

Example output

{
  "hands": [
    {
      "label": "left",
      "fingers": [1, 0, 0, 0, 0],
      "pointer": null,
      "gesture": null
    },
    {
      "label": "right",
      "fingers": [1, 1, 1, 1, 1],
      "pointer": { "x": 0.6839850544929504, "y": 0.8850594758987427 },
      "gesture": "swipe_left"
    }
  ]
}

The stream is JSON Lines: each emitted line is one complete JSON object. The current schema is additive and remains compatible with consumers of the original fields:

• schema_version: currently 2.
• sequence: increasing processed-frame sequence number.
• timestamp_ms: wall-clock capture time in milliseconds.
• hands[].pointer: pose-dependent cursor/control position.
• hands[].finger_confidence: per-finger extension confidence in thumb, index, middle, ring, pinky order.
• hands[].motion: pose-independent palm motion, including velocity (vx, vy, speed), newest displacement (dx, dy), and elapsed sample time (dt). Velocities are normalized units per second.
• hands[].gesture: a completed discrete gesture. A physical return stroke is ignored until the hand becomes still, preventing an opposite action.
• hands[].gesture_confidence: confidence derived from direction angle, trajectory straightness, and displacement.
• hands[].pinch: scale-normalized thumb/index pinch information. phase is one of idle, start, hold, or end. Change mode emits pinch transitions; continuous mode emits every held frame so the pointer can drive dragging.
• gestures: synchronized multi-hand gestures. Each entry includes type, phase, participating hands, and confidence. For compatibility, compound gestures also set the corresponding cardinal gesture on both hands.

An empty hands array is emitted when the last tracked hand disappears, so consumers can reliably release active controls.

Detected movement gestures

Single-hand swipes support eight direction sectors:

• swipe_left, swipe_right, swipe_up, swipe_down
• swipe_up_left, swipe_up_right, swipe_down_left, swipe_down_right

Cardinal swipes are enabled by default. Diagonals use narrower angular sectors and must be enabled with --diagonal-swipes. Movement in the ambiguity gaps does not emit a swipe.

When two hands are visible, Fingertrack uses their shared trajectory to detect:

• expand: hands move horizontally apart.
• contract: hands move horizontally together.
• push_down: both hands move down.
• pull_up: both hands move up.

Two-hand gestures share the same displacement threshold as single-hand swipes. Their return movement is suppressed until both hands have become still.

Command-Line Arguments

Fingertrack supports a few runtime options to tweak gesture detection and hand processing behavior:

--frame-skip N

• Description: Only process every Nth frame from the camera feed.
• Default: 1 (process every frame)
• Pros: Reduces CPU usage on slower machines.
• Cons: Skipping too many frames may make gestures less responsive.

--buffer-size N

• Description: Number of historical positions tracked per hand for smoothing gestures.
• Default: 5
• Pros: Larger values improve stability and reduce false positives.
• Cons: Too large may delay gesture recognition slightly.

--gesture-threshold N

• Description: Minimum normalized palm displacement required to complete a swipe.
• Default: 0.1 (approximately ten percent of the frame).
• This option is retained for compatibility with embedded consumers such as lifx-force.

--diagonal-swipes

• Enables diagonal swipe events.
• Diagonals use stricter angular confidence than cardinal movement and are disabled by default while camera-specific calibration is performed.

--consumer [stdout|http]

• Description: Select which output consumer to use.
• Default: stdout
• Options:
  • stdout: Print gesture events as JSON to the terminal.
  • http: Send gesture events to an HTTP endpoint.

--url URL

• Description: URL for the HTTP consumer when --consumer http is selected.
• Required if --consumer http is used.

--preview-mode

• Description: Controls the OpenCV hand-tracking preview window.
• Options:
  • full: Shows the camera feed with hand landmarks and overlays.
  • landmarks: Shows only hand skeletons on a black background.
• Default: Disabled
• Pros: Useful for debugging and visual verification of hand/gesture detection.
• Cons: Consumes additional CPU/GPU and is not intended for production use.

The preview identifies the palm anchor in yellow and the pose-dependent pointer in magenta. A green arrow shows palm velocity, and the thumb/index line turns yellow during a pinch. Text overlays show stable fingers, pose, swipe candidate, recognizer phase, pinch phase, strength, FPS, and completed two-hand gestures.

--record-jsonl PATH

• Records every processed observation rather than only state changes.
• Includes all 21 normalized landmarks for each hand so real camera failures can be inspected and converted into replay tests.
• Use --consumer none when a recording should not also be printed to stdout.

--output-mode [changes|continuous]

• changes is the default application-facing stream.
• continuous emits every processed observation and is intended for diagnostics and analogue controls.

For a text-only diagnostic session:

python app.py --output-mode continuous --preview-mode landmarks | python diagnose.py

The Vim pinch/drag POC also requires continuous output:

python app.py --output-mode continuous | python mime_poc.py

For a replayable recording without stdout noise:

python app.py --consumer none --preview-mode full --record-jsonl session.jsonl

Replay the recorded landmarks through the current detection code at their original timing:

python replay.py session.jsonl | python diagnose.py

Usage Tip:

• On a fast machine, you can reduce frame-skip to 1 for maximum responsiveness.
• For smoother gesture detection in a noisy environment, increase buffer-size slightly.
• Adjust threshold based on how sensitive you want swipes to be; balance responsiveness vs accidental triggers.

Docker

You can run FingerTrack either with a live webcam (Linux only) or with a video file (all platforms).

Linux (with webcam)

If you’re on Linux and have a webcam available at /dev/video0:

docker run --rm --device=/dev/video0 -it alessi0/fingertrack

This will start the app and stream from your webcam.

macOS / Windows (with video file)

Docker Desktop on macOS/Windows does not expose /dev/video0. Instead, you can run the app using a sample video:

docker run --rm -it -v $(pwd):/data -w /data alessi0/fingertrack python app.py --video myclip.mp4

Replace sample.mp4 with the path to a local video file. (You may need to mount your file into the container, e.g. -v $(pwd):/data -w /data.)

🖥️ Native Builds (macOS / Windows)

For users who want live webcam support without Docker:

macOS

Prebuilt binaries available for:

• amd64 (Intel Macs)
• arm64 (Apple Silicon)

1. Download the appropriate binary from the GitHub Releases
2. Make it executable and run:

chmod +x fingertrack
./fingertrack

• OpenCV will access your Mac camera natively via AVFoundation.

Windows

• Prebuilt binary (.exe) available for amd64 Windows systems.
• Download the executable from GitHub Releases and run it directly.
• Webcam access is handled via DirectShow.

Notes

• These native builds are fully self-contained — no Python or dependencies required.
• Use Docker only if you are on Linux or an ARM device and prefer a containerized environment.