TEI MI: AI— Building our Instrument, using light sensors!

Follow-up from last post, after the workshop with Daniel, we coupled our Arduino board for it to send serial inputs via processing to Wekinator. First we tested on potentiometers, then Melika or Josefine came up with the idea of using light sensors. We decided that light sensors would add a different dimension to the interaction— mostly because it affords richer interactions, in the sense that the one interacting with the computer would not be needing to physically touch any component from the Arduino board. This play on tangibility where you can physically manipulate dark and light— but also not touch anything— would be an interesting interaction to develop and work with.

We started with one light sensor. Just like the potentiometer, we first checked if the Serial to Processing file was able to detect the Arduino input that change the value, and if it did, we would proceed by experimenting the interaction with a pre-made output.

After testing with one and seeing it worked, we tried with two, so two hands could both be involved in the playing of the instrument. That worked too.

Josefine strongly advocated for incorporating an additional light sensor, the more the merrier— she said. By including another sensor, we’re opening up an opportunity for a collaborative musical interaction, involving more than one person.

Then it was time to ideate how the instrument would look! My first idea for our light sensor experiment was having a dark mystery box where you would reach your hand into the “unknown” to manipulate the music. The reasoning behind this was so that we could shine a light within the box so that the light source and its intensity would remain constant. But that was when we only had one light sensor in mind, if this box were to have multiple sensors, it would probably be difficult as an intuitive instrument, since you can physically “see” what’s going on.

Then Josefine proposed having three separate apparatuses that would contain the three sensors, we talked about more structure, boxed, rectangular prisms, but Melika and I mostly left the physical appearance of our instrument up to Josefine’s interpretation. I have good faith in Josefine’s prototyping skills so I was certain that she would create something we would all enjoy.

Whilst this was all going on, I was looking more into the code, tinkering and comprehending the code, and seeing if there’s any opportunity to switch out the music files so that we could also create our own songs in addition to simply creating a new interactive instrument. We were also influenced by our ambitious classmates with backgrounds in music production that were taking the drum machine example to the next level by adding their own beats and melodies. With a background in choral music and being part of choir since a very young age, I proposed swapping the .wav files into vocals, so we could somehow present a nice harmonization of different vocal parts and present that as a whole. Then, by manipulating the light sensors, we trigger the introduction of specific vocal parts. For example, we can train our program so that basses get introduced, then we introduce sopranos, then the tenors, and so on. Maybe even manipulate the sound by only having the sopranos and basses, the vice versa. At this point we haven’t thought too much about this yet, since I couldn’t figure out how to swap sound files in the All classifiers ChucK file.

Josefine created three cylinders of different sizes. These cylinders are capped with lids, so by lifting the lid, we introduce light in to trigger the light sensor, and vice versa. After hearing my idea of having different voice parts, she wanted to create cylinder jars of different sizes since her vision was that maybe we could have a deep sound, or a bass drum enter when the big jar was opened, then a high melodic sound be played when the thinnest cylinder was opened. She decorated these cylinders with traditionally woven ribbons that had tribal prints, so our instrument is not only functional, but also possesses an aesthetic appearance.

Inside these cylinders, we have our light sensors mounted at the bottom, attached to foam we found around the IOIO lab. We extended our wiring via longer wires also found around the lab. And ta-da! Everything works just as we imagined.