Piano Vision

Turn any piano video into a MIDI file. Drop a video, line up the detection dots with the keys, hit play, and export.

Built because I was tired of manually transcribing piano videos. This thing watches pixel brightness changes on each key and fires note-on/off events in real time. Not perfect, but surprisingly usable.

Live: piano-vision.pages.dev

---

How it works

1. Upload a video of someone playing piano
2. Drag the 88 dots onto the keys (drag-select groups, use the mapping controls to nudge/spread them)
3. Calibrate on a frame where no keys are pressed -- this captures the "resting" brightness
4. Hit play -- the detector watches each dot for brightness changes and records note events
5. Export MIDI when done

Each dot samples a 3x3 pixel area and compares average brightness against the calibration baseline. When the difference exceeds the sensitivity threshold, it fires a note-on. When it drops back, note-off. There's a configurable debounce to prevent rapid re-triggers.

Features

• 88-key detection with draggable, group-selectable dots
• Stretch handles -- select 4+ dots and drag the edges to adjust spacing
• Named presets -- save/load different dot layouts for different camera angles
• Sensitivity + debounce sliders for tuning detection
• Real-time note stream showing events as they fire
• MIDI export to standard .mid files
• Light/dark mode
• Onboarding walkthrough on first visit

Stack

• React 19 + TypeScript
• Vite
• Tailwind CSS v4
• Canvas API for frame processing
• midi-writer-js for MIDI export
• Deployed on Cloudflare Pages

Running locally

bun install
bun run dev

Or with npm:

npm install
npx vite

Limitations

• Detection is brightness-based, not ML -- works best with good lighting and a stable camera
• Doesn't handle camera movement or zooming during playback
• Black keys are harder to detect than white keys in most lighting conditions
• No velocity detection (all notes export at the same velocity)

License

MIT