September 5th, 2014

Improvising With Code

Kyle Stetz

As a creative person hanging out at the intersection of a bunch of things- design, programming, music- I'm always seeking new ways to mash them up.

I've had a long relationship with music and try every instrument and piece of software I can get my hands on. The deeper I go into the programming rabbit hole, however, the less I'm interested in creating the music itself and the more fixated I become on the tools. Tools are an inescapable part of music creation: there's a sense in which you are always either seeking or defying the sound of the instrument you are using. Experimental music (my cup of tea) tends toward the latter.

A new set of sophisticated audio tools called the Web Audio API now comes baked into modern browsers. Reading about it and seeing some preliminary examples got me excited, but I wasn't quite sure what to do with it. I have other audio software that is frankly much more powerful and less of a pain to use, so I waited for a moment of inspiration to strike... Which it did, in the form of this video!

The video is of Alex McLean's screen while performing live with a set of tools he made called Tidal- he's writing in the programming language Haskell. While the details of what he's doing are a bit complex, he is generating the music from the code he's writing. He wrote his PhD thesis on "Programming Languages for the Arts" and this has been the essence of his work for over a decade.

This video set off a chain reaction in my brain and connected all of my interests into one big complicated schematic. I love when that happens! I set out to translate this concept into Javascript and the browser. I'm very fond of the Google Chrome Console, which allows you to write Javascript in real time, so I decided that this would be the interface for my new tool.

Building an Instrument

When can we call writing code "playing an instrument"? What criteria have to be met? I wasn't quite sure at first, though I came up with a few guiding principles that helped frame the design of my Tidal/Google Chrome amalgam.

  • Provide logical (in the music theory sense) tools abstracted from the technical implementation
  • Provide the shortest path to a given operation
  • Support improvisation

That last point is a bit nebulous but ended up being a critical idea. What makes it possible to improvise with an instrument? Consistency, predictability, and maybe conciseness?

My contribution to the world of live music programming comes in the form of Lissajous.

Lissajous.js Performance

Over the course of four months I failed in building this tool twice, landing on an architecture I was happy with the third time around. The Chrome console demanded a different pattern of live coding: Tidal is built around an editable buffer of code that is constantly reevaluated, whereas Lissajous involves discretely executed code that manipulates a context. It forces you to keep more of the "state" in your head, which I might consider a disadvantage.

This project took me to a new level of programming ability and taught me some lessons too.


Improvisation means Flexibility

Writing an API usually involves a set of rigid (but predictable) rules that a programmer must follow. I did my best to ease the mental burden of remembering the details of 30 different functions by introducing flexibility. Almost any function in Lissajous can take any number of arguments (or no arguments!) and should result in predictable behavior. I tried to make the code fail as little as possible, as if to say "sure, you might have used a function incorrectly, but I won't yell at you because you are in the middle of something really intense!"


Planning for human error takes quite a bit of code because it means anticipating a lot of the assumptions and typos a person might make. Designing around such an extreme situation gave me a greater appreciation for the kind of code I want to write for my coworkers.

We never want to get rid of the possibility of error completely though... That's what makes a performance so exciting. Seeing someone weave a complex web of thought and sound and then somehow resolve it all through trickery of music theory or song structure without a single wrong note is to be taken on a tremendous journey.



Electronic musicians will bend your ear ad nauseum about the struggles inherent in performing live. The traditional interfaces you'll find in their toolbox are coated in knobs, sliders, buttons that light up in myriad colors– all to coax the sounds out of a computer. Just the same, popular audio software packages are designed around the controls you're likely to use the most, dismissing advanced features to hidden menus and panels. Mapping software to control surfaces is an art unto itself.

Improvisation implies a connection to an instrument so strong and direct as to make it disappear completely. Performing through code dispenses with (most!) physical limitations and presents all possible concepts equally, giving you an opportunity to make decisions on the fly by typing just a few characters. There are no "shoot which knob controls filter 2's cutoff frequency" moments.

As a fantastic byproduct, we can literally show the audience our screen and captivate them in a way we can't by finding stylish ways to twist knobs and push sliders.


Other javascript libraries like this are cropping up, some with incredibly advanced features like code sharing across multiple machines. I'm not sure that anyone else will ever use and love Lissajous, but it has certainly paid off in terms of my own work and perspective.

I used to get a little flustered when my coworkers would look over my shoulder and watch me write code. This is no longer the case, perhaps due to all the time I've spent programming frantically in time with music. I guess my job is just one big, complicated performance.

Grab the Lissajous codebase on GitHub or head over to to play with it right this second.

Check out another article
August 29th, 2014
Show, Don't Tell