I decided it was worth following hot on the heels of Photo Particles with another experiment. This time I really wanted to use audio as the input. Throw in some particles and some elastic mesh deformation and you're golden.
I'm all for using the right technology to solve the right problem, but I really love using some of the newer technologies over - say - Flash. I think it's actually because I don't have to recompile the code to see the changes. There are some things, however, that can't be achieved with WebGL or HTML5 right now. One of those things is audio spectrum analysis. Admittedly there is a little support for it in Firefox, but it's fair to say it's experimental.
I split the problem into two areas: audio capture and analysis and rendering. It turns out that although Flash can capture the audio from a microphone it can only perform Fast Fourier Transforms on sounds that are playing, which pretty much put paid to my plans of using Flash's native audio processing. Thankfully, Gerry Beauregard had written and released some audio processing code under an MIT license. Bingo!
After some monkeying around I had the Flash picking up the microphone audio and passing it out via External Interface to the browser.
From Audio to Awesome
I played around with the various mesh types and materials in three.js to get the right feel. Ultimately I wanted something that felt both organic and synthetic - quite a hard balance to strike, and I'm not sure I succeeded, but I settled on just a plain old sphere with some reflections.
With the sphere in place I mapped each vertex to the frequency buckets that Flash was passing through. I actually store two mappings; one that maps the y position value of the vertices to the frequency, and another that just picks a random bucket - both are pretty cool.
Finally it was a case of creating the particles. A pretty simple way of doing this was checking the magnitude of the incoming audio data and if it passes a particular threshold we create a particle at the vertex position. Net result: particles that are created from audio. How can you not love that?
I'd particularly like to thank Gerry Beauregard for his FFT audio analysis code, without which this experiment would've been much more difficult!