It was bring your kid to office day at TI and my daughter’s bday was coming up soon as well. I thought I should do something to spook a few kids with some magic. Given the choc-o-bloc schedule these days, I did not want to spend more time than 2-3 hours. Fortunately for me, my team at TI makes a secret ingredient for lots of magic – A 3D sensor.
I started with a OPT8241-CDK (Camera Development Kit). The kit provides a point cloud of 320 x 240 points. Point cloud is nothing but a collection of X, Y, Z and I (intensity) for each pixel in the camera. For example, this is what the point cloud of a lamp looks like –
After some thought, I decided to make a simple invisible musical instrument that even toddlers can appreciate (my daughter just turned 3). The 3D camera was to face up so that kids can move their hands over the camera to generate musical sounds. The distance of the hand from the camera would fix the amplitude and the lateral position of the hand would fix the note.
The code simply identifies the nearest blob in the scene and it’s 3D position. The ‘Z’ co-ordinate is used for the amplitude, The ‘X’ co-ordinate is digitized and used for the selecting the note. Since I decided to do everything in python, the midi-synthesis was just another line of import statement. All in all, as planned, I was able to complete the code under 2 hours. But the midi library was interesting and I ended up spending another 2 hours just playing with the various instruments available. Seriously, fluidsynth library is fun. The python code is hosted here. If you have an OPT8241-CDK with you and Voxel-SDK up on linux, all you need to do is run the python code.
The results (At my daughter’s school) :
2 thoughts on “The invisible harp!”
This is amazing, would like to try it out sometime!
Also just wondering whether ToF can detect velocity of the hand to decide the amplitude rather than the “x” position, that would be more realistic….
Without doubt, you’re paving the way to virtual music generation – playing music without instruments. I can imagine an orchestra with musicians, each one having only a CDK in front of them.
@Shantanu-ji, Did not expect anything less from you. Was expecting this “veolcity” comment to come from a musician. Next version perhaps.