This project focused on building a real-time gesture-controlled musical instrument that allows a user to "play" a piano using only hand movements. In a team of four, we designed a system where computer vision tracks both hands and converts finger positions into musical notes and tempo, which are then streamed to an ESP32 microcontroller for real-time audio generation.

The system maps the right hand to note selection (each finger corresponds to a note in a pentatonic scale) and the left hand to tempo control (BPM), creating an interactive, touchless instrument that responds continuously to live movement.

Design

The system was designed as a full real-time pipeline integrating computer vision, wireless communication, and embedded audio control. Python-based hand tracking detects finger positions and encodes them into a binary representation for notes, while also computing BPM from hand height.

This data is transmitted via UART to an ESP32, which interprets it to control audio playback. A GUI was implemented for key selection and system control.

Real-Time System

The ESP32 architecture was build around a hardware timer interrupt that served as a precise metronome for audio timing. Additional UART-driven updates handled finger states, BPM changes, key switching, and system status (e.g., no hands detected).

To ensure stability, we used interrupt prioritization, pre-generated audio samples, and hysteresis for BPM updates to reduce jitter and maintain consistent timing.