The sounds of my action — An interaction design project

Photo by Ahmad Odeh on Unsplash

The first step

I have a computer science background and some knowledge of music, but still, this kind of project was entirely new for me. The first step I did was to search for what are the requirements. On the hardware part, I thought of getting some sensor and I didn’t know which sensors to use. After some quick online search(asking dumb questions to google), I decided to use accelerometer and gyroscope sensors.


I was searching online for sensors and got an accelerometer. It wasn’t working well and I found apparently, it’s a low-quality one(poor me😔). The good ones are too costly for me and exceeding my budget. But, I remembered that these sensors are available on my phone too. I just installed some app to check whether they give the values the way I want them to. It was good, the values were exactly what I needed for both accelerometer and gyroscope. Wow!!

Generating sounds from numbers😳

The kept my phone/sensor aside for some time(both figuratively and literally). I sat on my laptop to figure out a way to generate music from custom numerical inputs. This part too was tricky considering its a 2-week course and I have to exhibit my work but the end of that. So, I asked Google (once again) and it told me to use PureData (or just Pd) which is an open-source visual programming language for multimedia. This was my first time using a visual programming language and didn’t know this thing existed before that day. I started to generate music from basic oscillators from some examples online. Oscillators — Straight out the physics laboratory, these things basically generate sine waves it seems. Those curvy lines in graphs?? Still don’t remember? That’s fine. An oscillator is things that produce sounds(actually waves which sounds😅) whose frequency I could change! It sounded like “Whee Wheee Whhee Whaaaww” (based on the frequency). It seems that’s how a pure sine wave sounds like — Like a siren. Anyway, it’s my first win in this project. A tiny one, but still a win I would take. Excited!!!

Better sounds with better software

Then, I figured out to send the notes from PureData as MIDI(Musical Instrument Digital Interface) notes. This way, I can use any one of the sound editing softwares out there. They are called DAW(Digital audio workstation) — just a fancy nice name, DAW (duh). I tried to use these softwares, to produce sound from the MIDI notes. The digital format of musical notes is called MIDI(just like JPG or PNG for digital images, MIDI is for digital musical notes and can be played by digital/virtual instruments). But this needs an IAC(Inter Application Communication) channel which is basically similar to having wire connection between different software inside a same computer, in this case between PureData and Logic Pro X, a DAW software that I’m using for this project. As soon as I set up the IAC channel and mapping them with instruments in Logic Pro, the notes computed from PureData where received by Logic Pro and played in a virtual instrument. That sounded far better. Like way better!! I started crying as soon as I heard it. It was soooo goooooood!!! You should have seen me at that time. I was jumping around the room.

Continuing with phone

So now, I have an algorithm to convert some numbers into sounds. But yet, getting those numbers from sensors in a phone is a task I haven’t worked on. So, the first thing I did was writing a python server in my laptop to get sensor values and sending it to PureData. Why Python? Because I’m good at that and used the simple server setup for some other projects.

Finishing off

Finally, all was working. Sensor values from phone to python. Python to PureData. PureData sends computed MIDI notes to Logic Pro X. Logic Pro X plays those in a virtual instrument. But the indented work of PureData wasn’t what it was doing. So, I simply removed it and wrote the same logic that it was performing in python itself. One less thing to worry about. Sigh.

A basic diagram representing how the whole thing works

The exhibit😬😎

Was the exhibit exciting? You bet. It was sooooo exciting!!!! There were all kinds of displays by many people on a range of electives. And mine too was there😁😎. With my exhibit. I had kept my phone in front of my laptop and speakers. People could lift it up and start to move it around to produce sound. Believe it or not, people just loved it. Some even took my phone and started dancing around the place. Yeah, you can say it was worth the effort working those two weeks. Felt so nice.

A demonstration from the exhibit

Going forward…

What is next after this? I have given it some thought lately. The first thought I had was a band with sensors on both hands. So, the hands will be free and two hands can create two different sounds at the same time. But later, I got to know about Kinect Sensors and similar ones. Kinect is a line of motion sensing input devices produced by Microsoft. It makes mapping the space and tracking the human skeleton easier. So, the next thing is that. No phone, No band, just a sensor at a distance to track the hand movement using motion tracking and generate sounds from that. Let’s see. I’ll post a continuation to this story once I complete that story.


A mobile app to get and send sensor values to a python server running in a laptop in the same network as the mobile. The python server processes the data from the mobile phone using mathematical functions and generates a MIDI pitch (musical note). It is sent to an IAC channel. This can be accessed by any musical software to play sound. In this case Logic Pro X, a DAW is used to play the generated notes.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store