this is really cool. i especially like the gesture mapping... using the distance between your hands for volume, their average height for pitch, and the angle between them for vibrato feels unusually intuitive. the fact that those dimensions can move independently makes this feel like it could support a lot more than a theremin sound
they're LLM-generated, web-embeddable synths, and the real-time controllable parameters can be defined in the prompt — so your hand tracking could map directly onto whatever controls make sense for the instrument
Interesting product. Are you really using Fable 5 to generate sounds?
FYI, in Chrome for Android, the only way to hide those bottom sheets is to tap the handle. Grabbing and sliding them down, reloads the page, which is a little annoying.
I created a gamelan sequencer with it. I intended on using samples but it went ahead and created a bunch of synthesized instruments in javascript with the right (ish) overtones, in the right tuning system, it was pretty crazy. 3d instruments animation with 3d audio positioning.
I’m exploring an analogous idea for music. The quality of the musical output has improved noticeably with the latest frontier models
Karpathy’s point about the models not being able to easily audit their work is something I’m struggling with —- how can the audio output be made perceivable? Curious what people think about this question.
i've spent the last couple years experimenting with LLMs that write procedural, ambient music: instead of generating an audio file, we're using systems that write actual source code (supercollider synths) so the music runs live and can be embedded in and controlled in real time by software like games. a surprising amount of the work is taste -- prompt engineering and curating an aesthetic so the generated code reliably compiles and sounds good, with a compiler and automatic repair in the agent loop when it doesn't build. today, it produces single-synth musical systems, but I'm researching how to orchestrate multiple synths into fuller, evolving compositions with deep musical form and structure. LLMs make algorithmic music far more accessible to artists, and I'm eager to hear the new kinds of music that come out of it. hit me up if you're an audiophile interested in collaborating.
a little suite of word games I've been building as the testbed for Underscore. every puzzle is generative: crosswords with LLM-written clues, a word ladder, a guess-as-it-draws game, and a deduction game scored on how few clues you need. the crossword grids come out of a Rust constraint-satisfaction solver — backtracking fill over a bitmask-indexed word list, enforcing 180-degree rotational symmetry and slot intersections. each generated puzzle gets its own generated, continuously-evolving soundtrack with no audio files involved. right now, the music is generated per puzzle, but it needs some work. it's currently too drone-y, and needs to be more magical/whimsical. where I want to take it is having it respond to the gameplay itself — which is really the whole point of Underscore
I’m working on Underscore: a web SDK and browser sandbox for making procedural music systems with LLMs.
The basic idea is "music with source code." Instead of prompting for finished audio files, you use an LLM to help write and revise a SuperCollider-based system that runs in the browser via WebAssembly [1]. The result is executable music: inspectable, editable, versionable, and controllable at runtime.
I’m especially interested in adaptive sound for software: games, creative tools, meditation apps, AI agents, interactive art. Places where a static audio file feels too dead, but hiring a composer/sound designer for every variation is unrealistic.
It’s early, but the thesis is that LLMs make algorithmic music much more approachable because code becomes a conversational medium. I wrote a longer piece about the idea here: https://x.com/osetinsky/status/2053674503801028944?s=20
- Sam Aaron for building SuperSonic, allowing for SuperCollider in the browser as an AudioWorklet: https://sonic-pi.net/supersonic/demo.html. Earlier, pre-LLM versions of Underscore relied on low-latency WebRTC implementations for streaming SC synths running on servers to browsers in real-time
Very half baked, even took that demo site featured in the video down. Intended to run hundreds of "channels" of music, each of which could support thousands of independent sessions (think cloud gaming but for audio/music).
Uses Pion WebRTC, Gstreamer, JACK, SuperCollider and a simple Golang API for converting http requests to OSC toggles to manipulate audio running on a cloud server in realtime over the internet.
Questions – is this crazy? Would game makers ever be interested in adopting a third party SDK for their music needs? What if that music had to be streamed over the wire (not baked into their console/mobile client builds)? Is the prospect of game player input influencing the sounds/music they're listening to compelling, or not really?
also - i made you some sounds with Underscore:
chill / evolving harmonic thing: https://underscore.audio/s/cmp_f2b79a32-57e/chord_lantern
chaotic / glitchier thing: https://underscore.audio/s/cmp_6abde2c8-2d9/morseMind
they're LLM-generated, web-embeddable synths, and the real-time controllable parameters can be defined in the prompt — so your hand tracking could map directly onto whatever controls make sense for the instrument
you can also generate more structured compositions like this: https://underscore.audio/s/cmp_9175f7d2-fe0/drifter_glimmer
happy to help you wire one up / iterate on some sounds if you're interested
disclaimer: i'm building Underscore :)