Every Gesture Synth Hand Sign, and Exactly What It Plays

Everything below was read off the instrument's own guide at gesture-synth-weld.vercel.app on 28 October 2026, not from secondhand write-ups. That matters, because the most-cited published description of the right hand is wrong - see the last section.

Before you start: two settings and a mode

The app opens on a start screen with two buttons, "📖 learn how to play" and "🤘 play now", and a quick tutorial. Camera access is required; the page says so plainly, and until it sees you it displays "waiting for your hands...".

Pick a playing mode first. There are two: Two-Handed and Left Hand Only. Left Hand Only is the one nobody notices and the one to start with - it drops the right hand's inversion, octave, volume and filter layer entirely, so you learn the seven chord degrees without fighting four other controls at once. Switch to Two-Handed once the degrees are automatic.

Then set the key. Twelve buttons cover A, A#/Bb, B, C, C#/Db, D, D#/Eb, E, F, F#/Gb, G and G#/Ab. The degrees are relative to whichever you pick, so a I chord in C is C major and a I chord in F is F major.

Five sound presets: Warm Synth, Bright Synth, Retro Synth, Strings 🎻 and Horns 🎺. Warm Synth is the most forgiving for learning because its attack is slow enough that a mistimed chord change does not click.

Left hand: the chord

Tilt decides quality: inward for major, outward for minor. The display confirms it visually - major chords render at full opacity, minor chords at 45%, so you can see which one you played without listening for it.

Pitch shift by height

The left hand also carries a semitone shift, but only after you enable "Raise" or "Lower" under Pitch shift. Once enabled, the screen splits into three zones: the top 50% shifts up one semitone, the middle 30% plays normally, and the bottom 20% shifts down one semitone. The asymmetry is deliberate - the normal zone sits where your hand naturally rests, so you have to reach to bend a note.

Right hand: the shape

The thumb is doing double duty, and this is the single most awkward part of the instrument: the same hand that counts 1-4 fingers for voicing also signals octave with the thumb. Counting to four with the thumb out means a first-inversion chord an octave down whether you wanted the octave or not. Decide your octave first, then count.

What changes under the hood

Two numbers explain why the instrument feels smooth rather than stepped. Volume changes ride a 0.03-second time constant and filter changes a 0.04-second one, so the parameter glides to its target instead of jumping. Chords are also held for a fixed interval before being released, which is what stops a momentarily misread hand from cutting the sound.

The tracking runs MediaPipe Hand Landmarker via WebAssembly on a 640x480 camera feed, following two hands at 21 landmarks each. Audio is Web Audio API: oscillators into a four-times oversampled waveshaper, then a biquad lowpass filter starting at 1200 Hz with Q 0.7, then a master gain. All of it runs in the page - nothing is uploaded.

Where published descriptions get the right hand wrong

Coverage from August 2026 describes the right hand's four finger configurations as selecting major, minor, dominant and diminished chord quality, and describes left-hand quality as coming from the middle-finger knuckle position relative to the wrist. Neither matches the app today: the right hand sets inversion and seventh, and quality comes from left-hand tilt. The same article states both the knuckle rule and a tilt rule without reconciling them.

Some of this is the app having moved on - the repo had 88 or so stars in August and 147 now, so it has been worked on. The practical rule: trust the in-app guide, reachable from the "Open Guide" button and the "?" control, over anything written about it. If a tutorial you find describes right-hand fingers as major or minor, it is describing an older build.

FAQ

Why does Gesture Synth play nothing when my hands are visible?

Both signs have to be recognised at once in Two-Handed mode. A partly occluded or ambiguous hand produces silence rather than a wrong chord, which reads as the app being broken. Switch to Left Hand Only to confirm tracking works, then go back. If there is no sound at all in either mode, that is a separate audio problem - the fix is in our Gesture Synth setup and no-sound guide.

How do I play a minor chord?

Tilt the left hand outward. Inward is major. You can confirm it on screen - minor chords are drawn at 45% opacity against full opacity for major.

Can I change the volume with my hand?

Only after switching "Volume: Fixed" off. The same applies to "Filter: Fixed". Both default to Fixed, which is why right-hand height and tilt appear to do nothing.

What are the two playing modes for?

Two-Handed uses both hands for chord and shape. Left Hand Only drops the right-hand layer, so you play degrees I to VII and nothing else. Start there.

Which key should I pick?

C if you want the degree table above to read as plain note names. Any of the twelve works identically - the finger mappings are relative to the key, not absolute pitches.

Does it record what I play?

The app as it stands on 28 October 2026 exposes modes, presets, keys, pitch shift and the two Fixed toggles. It shows no record or export control, so capture it with screen or system audio recording if you want to keep a take.

Sources

Gesture Synth in-app guide, read 28 Oct 2026 · ericwei97-cloud/gesture-synth on GitHub · DeepWiki architecture notes · A2A Protocol, 16 Aug 2026 (the mapping this article corrects)