Quick answer - the repo, the commands, the licence
Source: github.com/ericwei97-cloud/gesture-synth - 147 stars, 54 forks, 14 commits
Stack: JavaScript ES modules, MediaPipe Hand Landmarker, Web Audio API, built with Vite, hosted on Vercel
Run it: npm install, then npm run dev, then open the Vite URL and allow the camera
Build it: npm run build
Licence: NOT open source - a custom grant for educational and non-commercial use with original credit retained
Deploying it: camera access requires HTTPS in production; localhost is exempt
Gesture Synth is one of the few AI-driven instruments whose whole implementation you can read in an afternoon. Fourteen commits, no backend, no model to download at build time - MediaPipe's hand landmarker loads in the browser as WebAssembly. That makes it a genuinely good thing to fork and learn from. It also makes the licence worth reading before you do, because "on GitHub" and "open source" are not the same thing here.
Running it on your own machine
Clone the repository, install, and start the dev server. The project is a standard Vite app, so the two scripts you need are the Vite defaults:
git clone https://github.com/ericwei97-cloud/gesture-synth.git
cd gesture-synth
npm install
npm run dev
Open the URL Vite prints, allow camera access, and the audio context starts on that same click. That ordering is not incidental - browsers refuse to start audio without a user gesture, so the camera permission click is what unlocks sound. If you wire up your own start button, keep audio initialisation inside the same handler or you will ship a silent instrument.
For a production bundle, npm run build emits to dist. Serve that directory from anything static.
The deployment trap: getUserMedia only works in a secure context. On localhost you are exempt, so a build that works perfectly in development will fail to see the camera the moment you put it on a plain HTTP host. Deploy behind HTTPS or it does not work at all.
What you are actually reading
Four subsystems, and the boundaries are clean enough to swap any one of them. Hand tracking is MediaPipe Hand Landmarker configured for two hands at 21 landmarks each. Gesture recognition sits on top and includes state stabilisation - the reason a momentary misread does not cut your chord is a hold interval, exposed in the code as a chord-hold constant. The music theory layer turns a recognised sign into a chord against a scale table. Synthesis is plain Web Audio: oscillators into a four-times oversampled waveshaper, a biquad lowpass starting at 1200 Hz with Q 0.7, then master gain.
If you want to change one thing and hear the difference, change the filter. Volume glides on a 0.03-second time constant and the filter on 0.04; widening those is the quickest way to understand why the instrument feels smooth rather than stepped.
The forks have diverged - check before you copy commands
There are 54 forks and they are not all the same project any more. Commands from one will not necessarily work in another.
| Repository | What it is | Stack difference |
|---|
| ericwei97-cloud/gesture-synth | The original, by Eric Wei | Plain JS ES modules, Vite, Vercel. npm run dev / npm run build |
| almendron02/gesture-synth | A substantially rebuilt version | React 19, TypeScript, Tone.js, Netlify, Node 22. Adds npm run check, lint, typecheck, test:run with Vitest and Oxlint |
| thisIsDanielJin/gesture-synth | A different idea - a hand-tracked controller for music-production parameters | Tone.js proof of concept, with VCV Rack and Ableton on its roadmap |
| Jia1267/gesture-synth | Reworked mapping | Left hand sets the key, right hand plays Do through Ti |
| rainbow686/gesture-synth-weld | Fork advertising a hosted "Pro version" | Points at a separate paid domain |
If you want a TypeScript codebase with tests already wired up, almendron02's is the more comfortable starting point and it requires Node 22. If you want to understand the original idea in the fewest lines, stay on Eric Wei's.
The licence is the part people skip
Gesture Synth is public on GitHub under a custom grant, not an OSI-approved open-source licence. The terms as described in the project are that it is free to use, modify and share for educational and non-commercial purposes, provided original credit is maintained. Two consequences follow, and they are the whole reason to read this section.
First, a permissive-licence reflex does not apply. There is no MIT or Apache grant here, so you cannot assume commercial use is allowed because the code is readable. Anything revenue-generating - ads, subscriptions, a paid hosted version, bundling it into a commercial product - sits outside the stated grant.
Second, credit is a condition, not a courtesy. A fork that strips the attribution is outside the terms even for non-commercial use. Notably, the live app's own footer credits "made by @indecisive.eric" and its guide names Eric Wei, while the hosted copies circulating under names like gesturesynth.art, gesturesynth.net, gesturesynth.me and gesture-synth.com vary in whether they carry that credit at all - and at least one fork advertises a paid hosted tier.
Verify before you rely on this. The characterisation above comes from the project's own README summary and third-party documentation of the repository, not from a full reading of the LICENSE file. Open LICENSE in the repository root and read it yourself before building anything commercial, and if your plan is commercial, ask Eric Wei. He is reachable through the site linked from the app's guide.
FAQ
Is Gesture Synth open source?
The code is public and forkable on GitHub, but the licence is a custom educational and non-commercial grant requiring credit, not an OSI-approved open-source licence. Public and open source are different things.
Can I download Gesture Synth as an app?
No. There is no installer or desktop build. It is a browser instrument - either use the hosted version or clone the repository and run npm run dev locally.
What do I need to run it?
Node and npm, a webcam, and a modern desktop browser. The original needs no API keys, no environment variables and no backend - MediaPipe loads in the browser.
Can I sell a version of Gesture Synth?
Not under the terms as stated, which cover non-commercial use only. Read the LICENSE file and contact the creator if you intend to charge for it.
Why does my deployed copy not see the camera?
Because it is served over HTTP. Browsers only grant camera access in a secure context, with localhost as the development exception. Put it behind HTTPS.
Which fork should I start from?
Eric Wei's original to understand the concept in the fewest lines. almendron02's if you want React 19, TypeScript and a test suite already in place, which needs Node 22.
Sources
ericwei97-cloud/gesture-synth ·
DeepWiki on the original repository ·
almendron02 fork README ·
thisIsDanielJin fork ·
rainbow686 fork ·
Gesture Synth live app