What a hand-controlled instrument does
A hand-controlled browser instrument combines camera-based hand tracking with a music engine. The camera is used to estimate simple hand landmarks—rough positions for fingertips, joints, palm, and wrist—then the page maps those landmarks to musical actions. It is not the same as recording a video call or uploading a performance to a remote server; the frames are interpreted on your device and turned into control signals.
Gesture Synth is deliberately constrained so the first interaction is understandable: finger count chooses a chord from the active progression. Because the chords are selected from the same progression, you can explore combinations without needing piano technique or music theory first. For a beginner-facing instrument, fewer dependable controls are better than many fragile ones, so the app uses one to four raised fingers, a closed fist, and hand height rather than asking you to reproduce complex sign shapes.
How gestures become chords, energy, and visuals
Finger count works well for chords because it is visible, memorable, and easy to vary without changing your camera setup. It provides a small set of discrete states, which reduces accidental triggering compared with mapping every pixel of movement to a new note. Each state points to a different chord in a predetermined progression. The musical arrangement does the harmonic work in the background, while you decide when to move between chord states.
The musical response is intentionally not a raw one-to-one copy of hand movement. A chord selection can start or change harmony, while hand height adjusts intensity—lift your hand to brighten the sound and open up the drums, lower it for a softer feel. Visual trails and pulse effects follow the same performance state, helping you see the change you made. This design makes the result expressive without requiring you to control every note, drum hit, or effect parameter independently.
See the full gesture and chord reference →Try it now in the player →
Start in four steps
Use a current browser and a camera that another application is not already using. Good, even light and a simple background make recognition more reliable.
- Open the player and choose a music style.
- Allow camera access only when the browser asks for it.
- Hold one to four fingers toward the camera to trigger a chord.
- Raise or lower your hand to shape energy; close your fist to pause.
Why lighting and framing matter
The tracker needs a clear view of your palm and fingertips. Backlighting can turn the hand into a silhouette, motion blur can hide finger separation, and a busy background can make a fast gesture less distinct. These are recognition limitations, not a sign that the app is recording your video.
Sit about an arm’s length from the camera, keep the full hand in frame, and face a soft light source. If recognition becomes jumpy, pause between gestures instead of rapidly switching finger counts. When something still feels off, work through the dedicated setup and troubleshooting guide rather than changing many settings at once.
Set up lighting, framing, and fix common issues →Read the camera permission guide →
Use keyboard mode when a camera is not practical
Keyboard mode is a useful fallback for a desktop without a camera, a shared computer, or a situation where you do not want to grant camera permission. It keeps the same musical idea while replacing the camera input with keys and pointer controls.
Using the fallback is also a good way to check that audio works before troubleshooting the camera. If keyboard mode produces sound but hand mode does not react, focus on permission, lighting, framing, and competing camera apps rather than the music engine.
Learn the keyboard playing method →Start a camera-free session →
Know what stays on your device
Gesture Synth processes camera frames locally in the browser to recognize your hand. The live camera feed is not uploaded, stored, or transmitted by the core hand-tracking feature. The app requests video only; it does not need microphone access to create its synthesized sound.
For more detail about permissions and common camera issues, read the Camera Help page before enabling your camera on a new device, and review the Privacy Policy for the full data-handling description.