VisionDrive AI

Real-time computer-vision racing controller that converts two-hand gestures into a virtual Xbox controller.

Hold an imaginary steering wheel in front of your webcam and drive. Rotating the wheel steers, holding the driving pose accelerates, a left fist brakes or drifts, and a right fist fires nitro. Input reaches the game as a virtual Xbox 360 controller, so any title that accepts a gamepad accepts VisionDrive.

No hardware wheel. No keyboard. No neural gesture model — just landmark geometry, measured and filtered.

---

Demo / Overview

You sit in front of a webcam and hold both hands up as if gripping a steering wheel. VisionDrive tracks both hands in real time, works out which is physically your left and which is your right, measures the tilt of the imaginary wheel between them, and reads whether each hand is open or closed.

That interpretation becomes an analog steering value plus three on/off actions, which are pushed into a virtual Xbox 360 gamepad that Windows and the game treat as real hardware.

A compact 480×270 preview stays pinned above the game so you can see what the system sees without leaving the race.

Two modes:

• --dry-run — interprets everything, sends nothing. Use this for tuning.
• --gamepad — drives a real virtual Xbox 360 controller.

---

How It Works

Webcam
  |
OpenCV capture             measured backend selection (MSMF / DirectShow)
  |
MediaPipe Hand Landmarker  21 landmarks + world landmarks per hand
  |
Physical Hand Identity     physical LEFT/RIGHT, sticky over time
  |
Racing Control Engine      wheel geometry + open/closed classification
  |
RacingIntent               steering, accelerate, brake, nitro
  |
Xbox Mapping               the only place button names exist
  |
vgamepad / ViGEmBus
  |
Virtual Xbox 360 Controller
  |
Game

vision/ knows nothing about gamepads or racing. controller/ knows nothing about hands. racing/ speaks only in accelerate / brake / nitro and never names a button — only racing/mapping.py does that, which makes retargeting a new game a config edit rather than a code change.

Two ideas worth knowing before reading the code

Handedness is physical, never positional. VisionDrive feeds MediaPipe the raw camera frame, so processing coordinates never depend on a cosmetic setting. For the MediaPipe Tasks API fed an unflipped frame, the returned label already names the physical hand — "Right" means the user's right hand — so no swap is applied. This convention was established by physical camera measurement, not by reading documentation; the legacy mp.solutions.hands "assume mirrored input" rule inverts it and is wrong here. The preview may be mirrored for comfort, but that flip happens only at draw time. A physical RIGHT hand drawn on the left of a mirrored preview is still RIGHT everywhere in the code.

Identity is sticky. MediaPipe decides handedness independently every frame, so it contradicts itself when hands cross, touch, or are partly hidden. A racing game driven by one flipped frame would brake instead of accelerate. vision/identity.py keeps one tracking slot per physical side — two hands can never both report LEFT — and assigns detections by palm-centre continuity, hand shape, and label confidence. Continuity outweighs a single contradicting label; only a sustained contradiction (5 consecutive frames by default) triggers a deliberate self-correction. When the evidence genuinely conflicts, the hand is reported AMBIGUOUS rather than silently swapped, and the racing engine refuses to drive on it.

---

Controls

Physical action	Racing action
Both hands open	Accelerate + Steering
Rotate the imaginary wheel	Analog steering
Left fist, right open	Brake / Drift
Left open, right fist	Nitro + Accelerate
Both fists	Safe state (no input)

Steering and hand state are separate channels — closing a hand changes what the car does, never where it points.

LEFT hand	RIGHT hand	Steering	Accelerate	Brake/Drift	Nitro
OPEN	OPEN	active	ON	off	off
CLOSED	OPEN	active	ON *	ON	off
OPEN	CLOSED	active	ON	off	ON
CLOSED	CLOSED	active †	off	off	off
either side unresolved	—	active	off	off	off
pose invalid / hand lost	—	0.0	off	off	off

* configurable via racing.behavior.accelerate_when_braking, default on so a drift can be initiated under power. † configurable via racing.behavior.both_closed_steering.

Nitro never cancels acceleration. The car is already driving; the boost is layered on top. This has a dedicated regression test (test_racing_engine.py::test_nitro_does_not_cancel_acceleration).

Xbox mapping

Racing action	Xbox input
Steering	Left Stick X
Accelerate	RT
Brake / Drift	LT
Nitro	RB

Defined entirely in racing.mapping in configs/default.yaml. Bindings differ between games — verify yours before driving live (see Validate Virtual Controller).

---

Features

• Real-time two-hand tracking (MediaPipe Tasks Hand Landmarker)
• Physical LEFT/RIGHT hand identity, not positional
• Temporal identity stability across crossing, occlusion and label flicker
• True analog steering from both-hand geometry, not three discrete states
• OPEN/CLOSED gesture classification with hysteresis
• Accelerate, Brake/Drift and Nitro as independent channels
• Virtual Xbox 360 controller output via vgamepad / ViGEmBus
• Fail-safe neutralisation on hand loss or ambiguous identity
• Compact 480×270 gameplay preview
• Always-on-top Windows overlay that never steals focus from the game
• Fully configurable mappings, thresholds and smoothing via YAML
• Dry-run and live gamepad modes
• 394-test hardware-independent test suite

---

Architecture

src/visiondrive/
  app.py                  main loop, CLI
  config.py               validated configuration (nested, all sections)
  errors.py               actionable exception hierarchy

  vision/                 knows nothing about gamepads or racing
    camera.py             OpenCV capture, measured backend selection
    hands.py              MediaPipe wrapper + pure result conversion
    handedness.py         physical handedness decoding (verified convention)
    identity.py           temporal LEFT/RIGHT stability
    geometry.py           bbox, palm centre, shape, display mapping
    types.py              Handedness, HandObservation, FrameObservation

  racing/                 speaks only in accelerate / brake / nitro
    engine.py             hands -> RacingIntent, the state table
    wheel.py              steering geometry + neutral calibration
    hand_state.py         open/closed classifier with hysteresis
    filters.py            One Euro / EMA steering smoothing
    mapping.py            RacingIntent -> Xbox (the only place buttons exist)
    types.py              RacingIntent, HandState, ControlStatus

  controller/             knows nothing about hands
    virtual_gamepad.py    normalised pad + vgamepad backend
    types.py              Button, GamepadState, clamping
    testing.py            in-memory backend for tests and dry runs

  diagnostics/            presentation only, cannot influence control
    overlay.py            the vision HUD (debug mode)
    racing_overlay.py     the racing HUD + wheel debug visuals
    gameplay_overlay.py   compact picture-in-picture overlay
    window.py             preview window, Windows always-on-top placement
    metrics.py            FPS and duration meters

scripts/                  manual hardware smoke tests + model download
tests/                    hardware-independent test suite
configs/default.yaml      validated defaults

The separation is enforced by tests, not just convention: tests/test_gameplay_overlay.py drives an identical scenario at several preview sizes and asserts every resulting RacingIntent is identical.

---

Requirements

• Windows 10 or 11 — required for virtual controller output only
• Python 3.11 recommended (3.10–3.12 supported)
• A webcam
• ViGEmBus driver — required only for real virtual-controller output

Verified on Windows 11 with Python 3.11.15, OpenCV 5.0.0, MediaPipe 1.0.0, NumPy 2.4.6.

The vision half — capture, tracking, identity, the racing engine, the preview and the entire test suite — runs without vgamepad or ViGEmBus. Only sending input to a game requires them.

---

Installation

git clone https://github.com/ahmedsayed1911/VisionDrive-AI.git

cd VisionDrive-AI

py -3.11 -m venv .venv

.\.venv\Scripts\Activate.ps1

python -m pip install -e ".[dev]"

Then download the MediaPipe Hand Landmarker model (~7.8 MB, not committed):

python scripts/download_hand_model.py

It lands in models/hand_landmarker.task. To keep it elsewhere, use --dest <path>, model_path: in configs/default.yaml, or the VISIONDRIVE_HAND_MODEL environment variable. If the model is missing, VisionDrive says so explicitly and prints the URL.

Confirm the install:

python -m pytest

Optional: virtual gamepad output

Real controller output needs two pieces — the vgamepad package and the ViGEmBus kernel driver that actually creates the virtual pad:

python -m pip install -e ".[gamepad]"

Read this before running that command. vgamepad ships as a source distribution whose setup.py launches the bundled ViGEmBusSetup_x64.msi during installation. Installing the package therefore starts a driver installation with a Windows UAC prompt. VisionDrive itself never installs drivers, silently or otherwise — the decision is yours.

To keep the two steps separate, install the driver yourself from the ViGEmBus releases page first, then install the package. Without either piece the vision stage still runs normally, and any attempt to use the controller fails with a message naming the missing piece and the fix.

---

Run

Dry run — interprets everything, sends nothing. Use this for tuning:

python -m visiondrive.app --config configs/default.yaml --dry-run

Live gamepad output — requires ViGEmBus:

python -m visiondrive.app --config configs/default.yaml --gamepad

Useful while tuning thresholds — prints steering, hand states and raw openness scores each time they change:

python -m visiondrive.app --log-racing --no-window --duration 30

Other flags: --camera-index N, --model PATH, --frames N, --duration N, --no-window, --display-mode gameplay|debug, --swap-handedness, --log-handedness.

---

Validate Virtual Controller

The mapping shipped in configs/default.yaml is a sensible default, not a verified binding for your game. Confirm it with the camera out of the picture.

First, with no driver needed at all:

python scripts/test_racing_gamepad.py --dry-run

Then for real. Open the Windows game-controller panel:

Start-Process joy.cpl

Leave it open, then run with a slow hold so each step is readable:

python scripts/test_racing_gamepad.py --hold 2.0

Xbox 360 Controller for Windows should appear while the script runs. With a race loaded, watch the car for each announced step: "accelerate" should drive, "brake / drift" should slow or drift, "nitro" should boost, and "accelerate + nitro" must boost while still driving — if the car coasts there, the throttle binding is wrong. If any step misbehaves, edit only racing.mapping and run again.

A broader controller test covering every button, both sticks, both triggers and value clamping:

python scripts/test_virtual_gamepad.py

Xbox 360 controllers report both triggers on a single shared Z axis: LT pushes it one way, RT the other, and pressing both cancels out. That is the driver's behaviour, not a bug in VisionDrive.

Only once the mapping is confirmed should you enable live CV control with --gamepad.

Verifying handedness on your hardware

Some webcams mirror in firmware, reversing the chirality of every frame. Measure rather than assume:

python scripts/debug_handedness.py

Hold up one hand you know the identity of. The script prints MediaPipe's raw label, the decoded physical hand, what the same frame flipped produces, and an independent anatomical chirality signal from the world landmarks. If Physical: names the wrong hand, set swap_handedness_labels: true in configs/default.yaml.

---

Gameplay Preview

Two display modes, switchable at runtime:

Mode	What it is	Use it for
gameplay (default)	480×270 picture-in-picture, pinned above the game	playing
debug	the full-size diagnostics window	tuning

display:
  mode: gameplay
  gameplay_width: 480
  gameplay_height: 270
  always_on_top: true
  position: top_right     # top_left | bottom_right | bottom_left | center
  margin_px: 15

This is presentation only. The camera still captures at camera_width × camera_height and MediaPipe still processes that full frame — only the rendered preview is downscaled, and only after tracking has run. Preview size cannot reduce tracking resolution or alter steering, which tests/test_gameplay_overlay.py asserts directly.

Runtime keys

Key	Action
g	toggle gameplay / debug preview
h	hide / show the preview
c	recalibrate the neutral wheel pose
q / Esc	quit

None of these touch the gamepad. Hiding the preview stops rendering entirely while capture, tracking and controller output continue unchanged.

Focus caveat. The overlay deliberately never takes focus — it is pinned with SetWindowPos(hwnd, HWND_TOPMOST, ..., SWP_NOACTIVATE) so the game keeps the foreground and every keystroke reaches it. The cost is that OpenCV only receives these keys while the preview window itself is focused; click it once first. Once the preview is hidden there is no window left to focus, so h is effectively one-way within a session — start in the mode you want via display.mode or --display-mode.

---

Configuration

configs/default.yaml is validated on load — an unknown key or an out-of-range value is an error naming the problem, not a silent default, and every problem in the file is reported at once.

Section	What it controls
camera (camera_index, camera_width, camera_height, target_fps, camera_backend)	capture resolution, rate and backend selection
racing.steering	dead zone, full-scale angle, response curve, invert, smoothing
racing.pose	hand spacing bounds, confidence floor, tracking grace
racing.hand_state	open/closed thresholds and confirmation frames
racing.behavior	which actions fire in which hand-state combination
racing.mapping	RacingIntent to Xbox inputs (the only place buttons are named)
identity	temporal LEFT/RIGHT stability tuning
display	preview mode, size, screen position, always-on-top

Two settings that are easy to confuse — they are strictly orthogonal:

• mirror_preview — display only. Flips the preview. Changes nothing about handedness, identity, or any semantic value.
• swap_handedness_labels — semantics only. Inverts MediaPipe's label. Leave it false unless measurement on your hardware says otherwise. Changes nothing about what the preview looks like.

Tuning

Symptom	Setting
Full lock needs too much arm movement	lower steering.full_scale_deg
Car twitches when holding straight	raise steering.dead_zone_deg
Steering feels laggy	raise steering.one_euro_beta
Steering feels jittery	lower steering.one_euro_min_cutoff
Hard to steer precisely near centre	raise steering.response_curve
Car turns the wrong way	steering.invert: true
Fists not detected	raise hand_state.closed_threshold
Open hands read as closed	lower hand_state.open_threshold
State flickers	widen the gap between the two thresholds
Control drops out constantly	widen pose.min/max_hand_spacing

How the numbers are derived

Steering geometry. Each hand's anchor is its palm centroid — landmarks 0, 5, 9, 13, 17 (wrist plus the four MCP joints). Those barely move when fingers curl, which is what makes closing a fist safe. Fingertips are never used.

angle = atan2(y_right - y_left, |x_right - x_left| * aspect_ratio)

Positive means the right hand is lower, which is what a clockwise turn does to hands at 9 and 3 — so positive is steer right. Using the absolute horizontal separation makes the angle immune to horizontal mirroring, so the steering sign is a physical fact rather than a display artefact. The dead zone is then subtracted, the remainder normalised over the full-scale span, shaped by the response curve, and smoothed by a One Euro filter that stays smooth at rest without adding lag during a fast turn.

Neutral calibration. Nobody holds an imaginary wheel perfectly level, so the driver's comfortable "straight" pose is measured rather than assumed: on activation, 400 ms of wheel angles are collected and reduced with a median, which ignores the odd mistracked frame. Press c to recapture it. Losing the pose for more than 2.5 s discards the neutral, since the driver has probably moved.

Open / closed classification.

openness = mean(|fingertip_i - wrist|) / |middle_mcp - wrist|

over index, middle, ring and pinky. The palm length is spanned by two joints that do not move when fingers curl, making it a constant ruler for that hand and cancelling hand size and camera distance alike. Four fingers are averaged so one occluded landmark cannot flip the state. The thumb is excluded — in a wheel grip it wraps the imaginary rim whether the hand is open or closed, so it is noise. A hysteresis band plus a 2-frame confirmation stops flicker at a cost of roughly 66 ms at 30 fps.

---

Performance

Measured on the development machine (Windows 11, USB 2.0 webcam, CPU inference):

Metric	Value
Capture	1280×720, 30.7 fps sustained
Vision latency	10.3 ms mean, 10.9 ms p95, 11.4 ms max
Racing control latency	0.017 ms mean, 0.041 ms max
End-to-end pipeline	30.7 fps with racing interpretation
World landmarks	present on every detected hand

Re-measured during the release pass on the same machine — 8-second bounded run, MSMF backend, no hands in frame:

Metric	Value
Camera negotiated	1280×720, 29.2 fps at open
Vision latency	15.6 ms mean, 17.0 ms p95, 18.1 ms max
Racing control latency	0.018 ms mean, 0.039 ms max

The racing engine costs well under 0.1% of a 33 ms frame budget — the control layer is free next to the vision stage. Vision latency and delivered frame rate vary with lighting, because webcams lengthen exposure in dim conditions and drop their frame rate to compensate. Measure your own with --duration N --no-window.

The loop is deliberately single-threaded — no queues, no multiprocessing, no neural gesture model. At 10–16 ms of vision work against a 33 ms budget, none of that is justified yet. diagnostics/metrics.py provides the numbers that would justify it later.

A measurement worth repeating on your own hardware: the capture backend matters enormously. On the development webcam, DirectShow ignored the MJPEG request, fell back to raw YUY2, and managed only 6–7 fps at 720p — uncompressed 720p30 does not fit in USB 2.0 bandwidth. Media Foundation negotiated a compressed stream and delivered 30 fps from the same camera. Rather than hardcoding a guess, camera_backend: auto measures each candidate backend at open time and keeps the first that reaches 60% of target_fps. If your camera behaves differently, pin camera_backend to msmf, dshow or any.

---

Tests

python -m pytest

394 tests, all passing — verified during this release pass, in 0.90 s. Every one is hardware-independent: no camera, no driver, no network.

Coverage includes handedness conversion, display mirroring, bounding boxes, palm centres, world-landmark passthrough, temporal association, hand crossing, duplicate handedness, ambiguity, controller clamping and safe cleanup, config validation, steering geometry and its invariances (translation, depth, finger flexion, mirroring), the neutral calibrator, the openness classifier and its hysteresis, the full four-way state table, the Xbox mapping adapter, and a safety suite covering hand loss, ambiguous identity, bad spacing, grace expiry and exceptions.

tests/test_handedness_pipeline.py traces a label end to end and guards against the inverted-handedness bug. Camera and real-driver validation live in scripts/ as manual smoke tests, since they need hardware.

---

Safety

VisionDrive fails to neutral, never to a stuck input.

• Hand lost — steering is held for pose.tracking_grace_ms (80 ms by default) to ride out a single dropped frame; throttle, brake and nitro release immediately. After the grace expires, steering goes to 0.0.
• Ambiguous identity — if the tracker cannot say with confidence which hand is which, the engine refuses to drive at all. It never guesses a side.
• Invalid pose — hands too close together, too far apart, or below the confidence floor deactivate control.
• Exceptions — any failure in the control path neutralises the pad rather than leaving the last state applied.
• Full state each frame — VirtualGamepad sends the complete state on every update, so a button no longer in the state is actively released. A stuck throttle is structurally impossible.
• Clean shutdown — the controller is reset and released on exit, including on Ctrl+C.
• Both fists is a deliberate conservative safe state: no throttle, no brake, no nitro.

---

Limitations

• Windows-only virtual Xbox output. The vision half is portable; ViGEmBus is not.
• Driver installation is your call. pip install vgamepad starts an MSI driver installer. VisionDrive will never do this for you.
• Webcam and lighting dependent. Poor lighting degrades detection — and lowers frame rate, because the camera lengthens exposure to compensate.
• Gesture thresholds may need tuning. The openness thresholds are derived from hand anatomy and verified synthetically; expect to adjust racing.hand_state for your hands and camera distance. Use --log-racing to read your own scores.
• Controller mappings differ between games. The shipped mapping is a default, not a verified binding. Confirm it with scripts/test_racing_gamepad.py first.
• CPU inference. No GPU delegate is configured. Fast enough at 720p30, but a slower machine may need 640×480.
• Two hands maximum, which is all a steering wheel needs.
• Identity is a heuristic, not a tracking model. Hands that overlap almost completely for a long time can still end up swapped; the tracker reports AMBIGUOUS while unsure and self-corrects on sustained evidence.
• Nitro is hold-semantics. A held right fist holds nitro. If a game prefers a tap, that is a mapping-layer change; the vision side does not move.
• This is an experimental computer-vision interface, not production automotive control. It drives video games. Nothing here is suitable for controlling a real vehicle or any safety-critical system.

---

Roadmap

• Verified mapping profiles for additional racing games
• Per-user calibration of openness thresholds instead of manual tuning
• A small number of additional gestures (handbrake, look-back)
• Preview customisation — themes, layout, opacity

---

License

MIT — see LICENSE.

MediaPipe and its Hand Landmarker model are Google products distributed under their own licenses; the model is downloaded at setup time and is not redistributed by this repository. ViGEmBus is a third-party driver by Nefarius Software Solutions, installed separately by the user.

---

Topics: computer-vision · hand-tracking · mediapipe · opencv · python · gesture-recognition · virtual-gamepad · racing-game · xbox-controller · real-time-ai