What a hand-controlled instrument does
A hand-controlled browser instrument combines camera-based hand tracking with a music engine. The camera is used to estimate simple hand landmarks, then the page maps a gesture to a musical action. It is not the same as recording a video call or uploading a performance to a remote server.
Gesture Synth is deliberately constrained so the first interaction is understandable: finger count chooses a chord from the active progression. Because the chords are selected from the same progression, you can explore combinations without needing piano technique or music theory first.
Start in four steps
Use a current browser and a camera that another application is not already using. Good, even light and a simple background make recognition more reliable.
- Open the player and choose a music style.
- Allow camera access only when the browser asks for it.
- Hold one to four fingers toward the camera to trigger a chord.
- Raise or lower your hand to shape energy; close your fist to pause.
Use keyboard mode when a camera is not practical
Keyboard mode is a useful fallback for a desktop without a camera, a shared computer, or a situation where you do not want to grant camera permission. It keeps the same musical idea while replacing the camera input with keys and pointer controls.
Using the fallback is also a good way to check that audio works before troubleshooting the camera. If keyboard mode produces sound but hand mode does not react, focus on permission, lighting, framing, and competing camera apps rather than the music engine.
Know what stays on your device
Gesture Synth processes camera frames locally in the browser to recognize your hand. The live camera feed is not uploaded, stored, or transmitted by the core hand-tracking feature. The app requests video only; it does not need microphone access to create its synthesized sound.
For more detail about permissions and common camera issues, read the Camera Help page before enabling your camera on a new device.