AI Virtual Painter & Canvas

Draw in the air with your fingertip. A computer-vision "virtual pen" turns the tip of your index finger into a digital brush that writes onto a live webcam canvas in real time, with a top toolbar to pick colors, switch to the eraser, adjust brush thickness, undo/save, and clear everything.

---

Features

• Contactless drawing – gesture-controlled pen, eraser, and tool picker.
• Clean overlay rendering – strokes live on a separate canvas and are fused onto the camera feed with bitwise masks (no visual noise, no background bleed-through) – even low-luminance colors like pure blue stay fully opaque.
• Smooth strokes – the pen cursor is low-pass filtered (EMA) before drawing, removing hand jitter; consecutive segments are connected, so fast hand movement never leaves broken marks.
• Two-handed drawing – both hands draw independently at the same time (handedness labels from MediaPipe keep each hand's state stable).
• Undo / Save – revert the last stroke or export the drawing as a white-background PNG with one gesture or keystroke.
• 12-slot toolbar – Blue / Green / Red / Yellow / Cyan / Eraser / Clear / Undo / Save / Size− / Size+ / live pen-size readout.
• Stable-click selection – a toolbar button fires only after ~0.7 s of steady hovering (or when you drop the fingers to "click"), so moving the cursor across the header never triggers accidental tool changes.
• Gesture state machine – automatic switching between Selection, Drawing, and Palm-Eraser modes, per hand.
• FPS overlay – smoothed live FPS + current tool / brush size in the HUD.

---

Gesture cheat-sheet

Gesture	Mode	Effect
Index and middle up	select	Cursor between the fingertips, pick tools in the header
Index up, middle down	draw	Draw with the currently selected color
Open palm (all fingers up)	palm	Quick eraser (thick black wipe)
Eraser tool selected	eraser	Precise erase with a wide brush
Clear All (button, gesture, or C)	—	Reset the whole canvas (and the undo history)

---

Toolbar

Row 1                  Row 2
┌────────┬────────┬────┐
│ BLUE   │ GREEN  │ RED│ YELLOW  CYAN  ERASER
│ CLEAR  │ UNDO   │ SAVE│ SIZE -  SIZE +  PEN 12px (read-only)
└────────┴────────┴────┘

Hover the two-finger cursor (the white crosshair) over a button: it highlights immediately with a cyan border and fires after ~0.7 s steady hover. Only the selected tool stays highlighted in cyan between visits.

---

System architecture

webcam
  │  (BGR frame, mirrored)
  ▼
MediaPipe Hands  ──►  HandTracker          (landmark 8 = index tip, 12 = middle tip)
  │
  ▼
Gesture state machine  ──►  SELECT / DRAW / PALM-ERASE   (per hand)
  │
  ├─────────────── draws on ──────────────┐
  ▼                                      ▼
camera frame                          NumPy canvas        (h, w, 3), np.uint8
  │                                      │
  └──────────── fusion ------------------┘
                      │
                      ▼
               bitwise masked overlay
                      │
                      ▼
                    display

1. Hand tracking (hand_tracker.py)

HandTracker wraps MediaPipe Hands and exposes:

• find_hands(frame, draw=True) – runs inference, optionally draws the 21-point skeleton;
• get_hands(frame) – a [{"label", "land"}] list for every detected hand: land is {landmark_id: [x, y]}, and label comes from MediaPipe's handedness classifier ("Left"/"Right"), so each hand keeps stable per-hand state across frames even when two hands are present (the painter draws with id 8, index tip, and id 12, middle tip);
• fingers_up(hand) – [thumb, index, middle, ring, pinky] booleans computed geometrically (tip above the pip joint), so it adds zero inference cost.

2. Digital canvas (main.py)

A parallel black matrix, same dimensions as the camera frame:

canvas = np.zeros((h, w, 3), dtype=np.uint8)

Drawing never touches the camera pixels – strokes are written with cv2.line(canvas, prev, tip, color, thickness) (anti-aliased), chaining the previous smoothed index-tip position to the current one so fast motion stays continuous.

Smoothing: before drawing, the raw tip position is low-pass filtered with an exponential moving average (SMOOTH_ALPHA = 0.35), which removes most hand jitter for clean, fluid lines.

Stroke model: every continuous pen run is recorded as {"pts", "color", "thick"} and committed to a history list when the drawing gesture ends, when the brush changes, or when the hand jumps farther than MAX_JUMP px. Undo pops the last committed stroke and rebuilds the canvas from the remaining history with cv2.polylines – so the drawing is fully reversible.

3. Frame blending (bitwise masking)

Each frame the canvas is fused onto the video:

gray = cv2.cvtColor(canvas, cv2.COLOR_BGR2GRAY)   # luminance of the drawing
_, ink = cv2.threshold(gray, 10, 255, cv2.THRESH_BINARY)         # any ink = solid
frame = cv2.bitwise_and(frame, frame, mask=cv2.bitwise_not(ink)) # erase video under ink
frame = cv2.bitwise_or(frame, canvas)                            # paint solid ink on top

Why this is clean: any non-black canvas pixel (threshold > 10) counts as ink, so even low-luminance colors like pure blue stay fully opaque. bitwise_and zeroes the video where ink exists (its inverse mask), and bitwise_or then writes the canvas pixels there – the strokes appear "on top" with zero background bleed, whatever their color.

The eraser is just a thick black brush: black pixels have zero luminance, so gray stays 0 there and the mask keeps the original scene – the stroke disappears as if erased.

---

Getting started

# 1. isolated environment
python -m venv .venv
.venv\Scripts\activate            # (Windows PowerShell)

# 2. install
pip install -r requirements.txt

# 3. run
python main.py

Camera window controls:

Key	Action
Q / ESC	quit
S	save drawing as painting_<timestamp>.png
U	undo the last stroke
C	clear the canvas
T, +, -	change brush size step by step

---

Project structure

AI-Virtual-Painter/
├── main.py            # main loop: camera, gesture state machine, canvas, blending
├── hand_tracker.py    # MediaPipe Hands wrapper + raised-finger classifier
├── requirements.txt   # opencv-python, mediapipe, numpy
└── README.md

---

Ideas / tuning

• Handedness of the thumb heuristic – fingers_up() uses an x-based thumb test that suits a mirrored front camera. It is not used by any gesture, so you can ignore it.
• Sensitivity – MAX_JUMP (150 px) controls how far the hand may jump before a new segment starts; raise it for very fast scribbling.
• Stability – STABLE_HOVER (0.7 s) sets how long a toolbar button must be hovered before it fires; raise it to be less trigger-happy.
• Brush sizes – edit SIZE_MIN, SIZE_MAX, ERASER_THICKNESS.
• More colors / tools – add entries to COLORS, ROW0, ROW1, and LABELS in main.py.

A learning-grade project: great for practicing MediaPipe, OpenCV masking, and simple state-machine UI design. Not production-graded graphics software.