AR Piano Tiles

A fully functional, real-time Augmented Reality rhythm game built in Python using OpenCV and MediaPipe.

This project tracks your hand movements through a webcam and allows you to play a falling-tiles style rhythm game by physically pinching or swiping in the air.

Screenshots

Features

• Real-time Computer Vision: Uses cvzone and MediaPipe for high-speed, accurate hand tracking without requiring markers or physical hardware.
• Hybrid Input Mechanics:
  • Swipe Mode: Hover and swipe your index finger in the lanes for rapid gameplay.
  • Pinch Mode: Use a thumb-and-index finger pinch gesture for precise note targeting.
• Dynamic Rhythm Engine: The game does not just spawn tiles randomly. It reads arrays of musical tuples (Frequency, Duration) and dynamically adjusts the physical spawn gap between tiles to perfectly match the rhythmic sheet music of the song.
• Custom Synthesizer Audio Engine: Instead of relying on pre-recorded MP3s or raw system beeps, the engine generates and plays its own high-quality .wav files using math-based sine waves, harmonics, and exponential decay envelopes for a rich, authentic piano sound.
• Included Songs:
  • Twinkle Twinkle Little Star (Easy)
  • Beethoven's Ode to Joy (Medium)
  • Beethoven's Fur Elise (Medium-Hard)
  • Tetris Theme - Korobeiniki (Hard/Fast)

Prerequisites

You need a webcam and a Python environment with the following dependencies installed:

pip install opencv-python
pip install cvzone
pip install mediapipe

How to Play

1. Clone this repository.
2. If the sounds/ directory is missing or empty, generate the audio files by running the synthesizer script:

   python generate_sounds.py

3. Start the game engine:

   python ar_piano_tiles.py

4. Stand in front of your webcam and raise your hand.
5. Use your index finger (or pinch) to navigate the carousel menu (Left/Right), toggle your input mode, and hit START.
6. Smash the black tiles as they fall down the lanes to play the song!