The body is the ultimate test of successful engagement with interactive systems as it adds more liveliness, vitality and pleasure into the design process.
Lecture
Jens’s lecture consists of breaking down the two papers by Loke & Robertson— Study of Dancers and Labanotation for Design of Movement- Based Interaction. The main point of the dancer is acknowledging the three perspectives of exploring movement. Before introducing the three perspectives, the concept of Kinaesthetics is introduced. Kinaesthetics, drawing from greek language origins, essentially means the aesthetics of movement. It relates to a person’s awareness of the position and movement of the parts of the body by means of sensory domains (Oxford).
The Mover belongs to the kinaesthetic perspective— it belongs to the sensory proprioceptors. It’s the first person— the one doing the movement’s narrative. It’s the movement itself and tend to be more “difficult” to articulate.
The Observer, to be simply put, is the person observing the movement. The observer does not feel what the mover is feeling. However, based on previous knowledge, the observer can make their own interpretation of what the action entails.
The Machine perceives the movement in the observer’s perspective, but it’s different. It views the body via coordinates or inputs— or whatever you’ve taught it to learn from the action.
The first image that came to mind when discussing the three perspectives was a tennis match, or basically any sports game consists of players and an audience. The players themselves would belong to the mover perspective, the observer being the audience, and the machine would be the hawk-eye system that tracks the trajectory of the balls played and displays a profile of its statistically most likely path as a moving image, or collects data of the player’s movements.
The second paper about Labanotation was basically a notation that Loke & Robertson have created to reproduce movements. They have created a new set of vocabulayr to help us pay attention to specific details in movement, or to better identify movement and be able to retell it. This will definitely come in handy when we choose our “gesture” to analyze.
He ends the presentation with the usual “process” graph— and he emphasized that for this third module we will be tinkering more than implementing (mentioned in previous entry).
Again, M3 is all about challenging convention about gestures and seeing how machine learning can be implemented in any way.
Set-up
After going through a series of issues for maybe three days just trying to get the installation to work, and three versions of the code later, we finally got the program to work avoiding any errors.
For the tinkering phase of our design process, we’re supposed to really delve into and explore the capabilities of our machine learning code. The code is capable of recording gestures based on the data coordinates drawn from the movement sensors (accelerometer and gyroscope) in our smartphones.
For the entirety of this afternoon we devoted into recording different gestures 20 times, training them, and seeing if the program can further predict what gestures we’re doing.
We noticed that shapes that are vastly different tend to not confuse the computer so much– for example, differentiating between a circle, a triangle, and the movement of the uppercut (yes, I really did this). The computer was able to differentiate these very well and predict each gesture at ease, however, once i incorporated more complexity, such as incorporating a square gesture and a basketball gesture, the computer got slightly bit confused, in the beginning, it would confuse the basketball gesture with the uppercut, the square with the triangle, but eventually, it just predicted everything as a circle. We don’t really know what this entails, and how to improve the prediction, which is perhaps training an extra 10 more recordings, but we will proceed that in a later on iteration.
Then we removed all our present data and played with something new, instead of shapes, I wanted to see if the computer could recognize alphabets that I’m trying to spell out. Well, first I thought of the YMCA dance, because it’s an extremely “gestural” dance, but i figured just doing the dance the way it is isn’t so “rich” in movement, and I was certain the program wouldn’t be able to decode what alphabet I’m gesturing just by “pointing” in different directions (when you observe the YMCA dance, you’re angling your arms a certain way for each alphabet to spell it out, not so rich in movement…).
So I decided to write the alphabets out (like, literally WRITE it), and see what the computer predicts because it would be pretty cool to spell things on the computer just by using gestures etc. Unfortunately it failed miserably as practically none of the right alphabets were predicted. Again, we didn’t really delve into why this is happening other than acknowledging that the code is obviously not perfect and cannot be 100% accurate.
Reflection:
Machine learning, a subset of AI, we can train a certain code with enough data for it to be able to predict specific movements without explicit instructions. But what does this all mean? After the tinkering we’ve been doing, it’s time to step back and ideate what gestures we find particularly interesting and analyzing that, finding design opportunities perhaps, and how we can utilize this machine learning technology to a greater extent.
This all sounds exciting but I guess the limitation of the module is the phone itself. At the end of the day, the phone is quite chunky and it’s one single object, we cannot attach it on multiple joints for it to recognize “richer” gestures, we cannot attach it all five of our fingers so it knows how many fingers are being held up. Therefore, as a prediction, I feel like this module could potentially be quite challenging. I guess we shouldn’t be thinking about implementation just yet and focus on finding an interesting gesture as part of an activity to kick-off.
Journal with me! 