Show n’ Tell thoughts…

Overall I think we did pretty well by structuring our presentation with our guide of a powerpoint presentation . It was easier to just go through our process structurally / in order of what we have done and our thought processes as we underwent multiple iterations. Additional media like gifs were also used as a visual-aid for showcasing our process of analysis and really make clear to the audience of what gesture we chose and how we got there.

Post- Show n’ Tell…

After the presentation we were presented with some valuable advice from the teachers. In general as a statement, he recommended the students that this module was crucial for us to learn how to navigate through uncertain territories. Using the phone as a limitation, it adds challenge because phone safety becomes something that we need to be considerate of— for example, we don’t necessarily need to hold onto it, we can open up and explore bodily attachments as well and figure out ways to not necessarily press the phone screen at all times. And lastly he said something that I feel like would’ve been valuable for the whole process if I had heard it since the beginning, and that is that Machine Learning helps you articulate movement. Of course Machine Learning is never precise and it can often be difficult for it to notice and detect the nuances but articulation is a nice way to put it because the computer simple presents what it’s been “trained” to say.

We were complimented for being articulate about our selected movement. Additionally, we were told that we did a good job with taking the whole module in a systematic approach though Clint gave me a comment about my pitch / presentation that I didn’t necessarily need to present in a temporal matter and could’ve gone straight to the point jump straight into the gesture we found interest and not the process we took to get there. That’s something I’ll keep in mind in future presentations for sure. And also, like I mentioned in the previous entry, they said they were disappointed that we stuck with the same motion just performed by two people at the same time instead of amping up to a new level and creating a new movement instead. Again, I didn’t really get that we were supposed to reinvent our gesture and create something new that could be crucial to movement design and Machine learning but of course I only came to this conclusion after contemplating about it in hindsight and it wasn’t really intuitive at all. Now that I know this I do agree that we could’ve taken it much farther as we basically only took one sitting with our social interaction phase and we could’ve discovered more if we spent more time ideating.

As we wrapped up the first two weeks of this module and settling on one motion, it was time to be adventurous and find a potential for a novel interaction / direction that involves our “YES!” motion.

The final version of code presented for us (gestures-ml 1.06) supports multiple devices for training, and the predictions are said to be significantly better. Since we’re two with only one mobile device each, we experimented with two phones. We first thought about how we interpreted the individual “YES!” motion as being also a gesture of hiding and moving out of the way and ideated about new potentials for design of interesting interactive gestures. That didn’t go far because we quickly found out that when two people are doing this gesture at the same time we’re not “hiding” so much anymore, as opposed to that, the action resembles “fighting over something” and being “proud” when performed together at the same time.

We first talked about actions / activities required by two people and is performed simultaneously by both (while facing each other) that involves our “YES!” arm curling motion. We talked about sawing, as it requires two people to work contemporaneously, pushing and pulling the saw from both ends without one overpowering the other.

Then thought about other activities that demand the same type of movement and watched tug of war videos. Tug of War a contest in which two teams pull at opposite ends of a rope until one drags the other over a central line (Oxford Dictionary). For this activity, when two sides are pulling with the same force, there is some sort of an equilibrium achieved and the teams end up in a “stand-still” state, however, when one overpowers the other, or when one team loses the same strength and forfeits then the other team breaks down and the pulling side wins. We were quickly drawn to this concept of equilibrium so we explored that movement a little, using long rubber bands often used in workout circuits.

We documented this movement and then discussed the felt qualities like our first iteration but unfortunately didn’t push any further with it (which we later discovered during show-and-tell was essentially the “goal” of this module— to explore and re-design new movements that can contribute to interaction design), instead, we tested the machine learning technology using this movement with the newly published code that can collect data from two devices and predict them both synchronously and even more accurately.

Felt Quality

• Tension
• Resistance
• Bracing oneself, digging into the ground to prevent loss in balance
• Involving whole body
• Fighting for something
• Different from one person protection, proudly claiming what’s yours
• Actually grasping
• Synchronization & Team effort
• Pulling from side to side 

From pulling to throwing- as you can see, as opposed to the felt qualities we discussed in the individual “YES!” movement, the felt quality here doesn’t involve hiding as much as facing each other— similar to direct confrontation. IT more closely related to proudly claiming what’s yours and fighting for it instead of the previous conception from our previous gesture of shielding and shying away. We thought this was interesting as a discovery as we wouldn’t have recognized these without physically performing them in the mover perspective and analyzing in-depth. It’s thought-provoking how the exact same gesture performed by two people instead could inflict such a different feeling.

Machine Learning

Together, we tried four different gestures for recording. The gestures were to be performed simultaneously. We pressed down to record our individual gesture (that’s part of a whole) at the same time and released our thumbs at the same time. Like all the other iterations we’ve explored machine learning code with, we recorded each combined gesture thirty times each then tested whether the code could predict the right gesture and accurately differentiate the four. We experimented with the rubber band to feel the tension and resistance but during recording the rubber band was omitted.

The four combined gestures consisted of:

1. Think sawing motion— one person has arm in “YES!” and the other has their arm instead— going towards the right, which we called sawRight
2. Same as the above, instead going towards the opposite direction— other person does “YES!” and vice versa— we named it sawLeft
3. Pull apart from each other— both perform the “YES!” simultaneously— motion of stretch rubber band (see below), we call it stretch
4. Releasing the band in a dramatic way— both extending arm towards each other, almost like a high five— think from “YES!” to a high five— we named it push

Wrapping Up

This was essentially our wrap-up for this module. We’ve found an activity, identified an interesting movement or gesture out of that general motion of activity, analyzed it thoroughly, discussed about felt qualities, tinkered with the machine learning code, and wrapped up by finally giving social interactions a try with the final version of the model code.

We didn’t push as far as inventing a novel social gesture, but overall, we did our iterations strategically in a structural matter, which really guided us in conducting a solid wrap-up. To sum it up, AI as a field in technology is steadily growing and it’s extremely appreciable to have such an opportunity to tinker with pre-made machine learning code. In addition to that, kinaesthetics and the art of movement is essential to movement design and movement design is a fundamental study of interaction design. Though the instructions provided in So ultimately I think it was a valuable experience for us to be given this opportunity of combining these two concepts into one and perhaps helped some of us at least discover new opportunities spark new ideas for ourselves in the future (perhaps for a thesis research etc).

The “YES!” motion was introduced in the last entry where I discussed how we discovered an interesting aspect from the mechanics of pitching.

The “YES!” motion consists of doing some sort of a swiping / swinging motion starting from your hand being at a distance from the core of one’s body and eventually pulling it in and contracting your elbows close to one’s body.

This motion is prevalent in pitching and especially for educating young and amateur pitchers. It’s important because when the arm collapses inward towards the body it’s aiding the swinging arm to not run into obstacles during the twist as well as making sure the balance remains at the center of the body to prevent a breakdown.

In addition to how important the motion is, we agreed that as an assistance movement, it’s often overlooked and when we watch a pitcher in a baseball game, you don’t really notice what’s going on with their non-dominant arm as the focus tends to be drawn towards the baseball itself. So it’s nice to find out how a seemingly non-significant motion has such a significant value.

Looking at our film documentation of ourselves pitching, we again did the same frame-to-frame breakdown and jotted down notes supporting the action.

Again, this was helpful for us to focus on the target area of our selected movement, and it’s helpful have this kind of reference to go back to whenever we need some sort of visual aid.

Following these analyses we proceeded with jotting down actions we believe resemble or consists of the YES motion, including:

• Apple picking
• Swiping a BIG screen
• Bicep curls / pull-ups (many workout routines involving the arm)
• “Come here!”— motion of inviting someone over
• Lifting something from the ground
• Rowing

Machine Learning

It was finally time to test with the machine learning technology!

We organized ourselves by collecting four sets of data, all “YES!” motions but from different directions:

1. Swing from right side to center then “YES!”
2. Swing from left side to center then “YES!”
3. Swing from bottom to center then “YES!” (curling motion)
4. Swing from top to center then “YES!” (apple picking motion)

Visual reference:

To test the technology, we recorded each gesture 30 times this time around, as discussed before, we thought about trying to train more data to hopefully enhance the accuracy. Then with improved code (after a series of panic and anxiety from errors) provided by the teachers, we put the data to test. We noticed again that the technology is able to distinguish the swings from the left and right sides— when the movement moves across the x-axis the technology can easily predict correctly, but again, when the x-axis remains unchanged but the coordinates are only changing on the y-axis, the prediction became iffy and inaccurate. When we tried it again making the curling and apple picking motions more dramatic and unique from each other the results were more or less completely accurate. We didn’t delve too much into this after the second iteration, however, it’s at least revealing to know that the technology works and the differentiation is somewhat successful.

Felt Quality

After the analysis and testing the technology for our “YES!” motion, we proceeded to discuss the psychological aspects of our movement. We’ve been paying a lot of attention the motor aspects of our movement and have somehow neglected the other yet almost more important aspect for this module, how it feels to perform this action.

The Felt Quality of a movement refers to the sensation or feeling in the body (Loke & Anderson 2010). Following the Felt Quality model again from Loke & Anderson’s paper for analyzing dancers and falling, we wrote down a list of what we felt when we performed the movement first-hand. It consists of:

• Expressing extreme content
• Becoming smaller
• Hiding / Shielding / Crouching
• Swinging
• Regaining balance
• Staying out of the way
• Dragging self towards object
• Keeping to oneself
• Grasping (but not actually grasping anything)
• Sense of completion
• Beginning of a run, but actually staying still
• Pulling in
• “I want this, but I can’t have it”
• Protecting, “my precious”
• Shutting down

This analysis was significant as it revealed a diverse range of understandings of the process and experience of the motion of saying “YES!”, it revealed that on the surface, the motion may appear to only be used to express contentedness. But when you simply look at the body language and ignore the social aspects and your preconception about the action itself, you may find yourself viewing the motion in a complete different way and a completely different attitude may be conveyed— such as hiding and shielding as opposed to cheering for being first in a race— they’re so different yet a similar body language can be seen!

We kick-started today by watching Drew Storen’s pitching video through and through, then, I had the idea of screen shotting the video frame by frame to analyze the movement and placement of his joints and limps, tracking the movement by second. In addition to it being overall a easier way to analyze the movement, it’s also a nice reference to go back to for whenever we run into obstacles and need to recalibrate.

The first draft of our frame analysis is simply drawing dots for all the important joints that are involved in this full embodied movement, then connecting them with lines to represent the limbs. This was just a nice reference and visual aid for us to see what’s going on with the limbs. We then cropped these frames again and made a little stop motion gif to see it in motion again. We thought this was important for us to analyze to notice the subtle nuances in the movement and by drawing these dots were only paying attention to the joints and not whatever else is going on in the frames.

When analyzing, we also used this video as inspiration for analysis when it came to using more technical terms for analysis— such as what it entails when weight is shifted from leg to leg. This video gives a more informative description of the scientific aspect of Drew Storen’s pitching mechanics. We used this video’s text pop-ups as notes to jot down on our frame analysis.

Below is the second draft of our Drew Storen pitching mechanics break-down. Inspired by how Loke & Robertson (2010) analyzed the falling motion in their paper “Studies of Dancers”.

We went from frame to frame, writing down what’s going down, what techniques he’s using, what each change in movement represented, and so on. The red arrows represent major shifts in movement, such as the leg placement as well as the body twisting that’s essential for baseball pitching.

Taking inspiration from Loke & Robertson’s Studies of Dancers: Moving from Experience to Interaction Design, we analyzed in an observer‘s perspective of how we think the motion of pitching (based on Drew Storen’s video) unraveled. We made a table to analyze the dynamic unfolding of the bodily movement of pitching. Though Loke & Robertson wrote about the motion of falling, however, in a first-person perspective— or the kinaesthetic perspective.

This is our table analyzing throwing (click to enlarge):

I thought analyzing it this way and breaking the movement down into three sections / increments was helpful for us to articulate what we’re seeing on the screen. It’s like utilizing a new vocabulary for describing movement.

Following this analysis we went out again to document ourselves performing the action of throwing / pitching. Of course, we may not have the best and most accurate form, but we tried to mimic Drew Storen’s pitching technique.

We finally discovered something we had an interest in. From now on we will refer to it as the “YES!” motion or the “YES!” swipe. The “YES!” motion is often introduced by coaches for young pitchers to learn how to engage their bodies during the pitch instead of just letting the arm hang low when the ball is tossed. We first learned about the importance and the existence of this so-called “YES!” motion from this video↑↑

It’s referred to as a YES! motion because this specific gesture is commonly used as a way of expressing extreme content.

See the next entry for a more detailed analysis of the “YES!” motion.

I personally think systematically breaking down the action of pitching was helpful in aiding us gather our thoughts onto one piece of paper. Prior to doing this we were still feeling quite wishy-washy about choosing the movement of pitching but after this analysis we felt significantly more certain. I’ve never gone so in depth with a movement before so breaking it down frame by frame really inflicted the feeling of experiencing the motion first-hand. In addition to that seeing the visual aid also guided us into learning the right form and performing it ourselves the right way to feel the action in the mover’s perspective.

The body is the ultimate test of successful engagement with interactive systems as it adds more liveliness, vitality and pleasure into the design process.

Lecture

Jens’s lecture consists of breaking down the two papers by Loke & Robertson— Study of Dancers and Labanotation for Design of Movement- Based Interaction. The main point of the dancer is acknowledging the three perspectives of exploring movement. Before introducing the three perspectives, the concept of Kinaesthetics is introduced. Kinaesthetics, drawing from greek language origins, essentially means the aesthetics of movement. It relates to a person’s awareness of the position and movement of the parts of the body by means of sensory domains (Oxford).

The Mover belongs to the kinaesthetic perspective— it belongs to the sensory proprioceptors. It’s the first person— the one doing the movement’s narrative. It’s the movement itself and tend to be more “difficult” to articulate.

The Observer, to be simply put, is the person observing the movement. The observer does not feel what the mover is feeling. However, based on previous knowledge, the observer can make their own interpretation of what the action entails.

The Machine perceives the movement in the observer’s perspective, but it’s different. It views the body via coordinates or inputs— or whatever you’ve taught it to learn from the action.

The first image that came to mind when discussing the three perspectives was a tennis match, or basically any sports game consists of players and an audience. The players themselves would belong to the mover perspective, the observer being the audience, and the machine would be the hawk-eye system that tracks the trajectory of the balls played and displays a profile of its statistically most likely path as a moving image, or collects data of the player’s movements.

The second paper about Labanotation was basically a notation that Loke & Robertson have created to reproduce movements. They have created a new set of vocabulayr to help us pay attention to specific details in movement, or to better identify movement and be able to retell it. This will definitely come in handy when we choose our “gesture” to analyze.

He ends the presentation with the usual “process” graph— and he emphasized that for this third module we will be tinkering more than implementing (mentioned in previous entry).

Again, M3 is all about challenging convention about gestures and seeing how machine learning can be implemented in any way.

Set-up

After going through a series of issues for maybe three days just trying to get the installation to work, and three versions of the code later, we finally got the program to work avoiding any errors.

For the tinkering phase of our design process, we’re supposed to really delve into and explore the capabilities of our machine learning code. The code is capable of recording gestures based on the data coordinates drawn from the movement sensors (accelerometer and gyroscope) in our smartphones.

For the entirety of this afternoon we devoted into recording different gestures 20 times, training them, and seeing if the program can further predict what gestures we’re doing.

We noticed that shapes that are vastly different tend to not confuse the computer so much– for example, differentiating between a circle, a triangle, and the movement of the uppercut (yes, I really did this). The computer was able to differentiate these very well and predict each gesture at ease, however, once i incorporated more complexity, such as incorporating a square gesture and a basketball gesture, the computer got slightly bit confused, in the beginning, it would confuse the basketball gesture with the uppercut, the square with the triangle, but eventually, it just predicted everything as a circle. We don’t really know what this entails, and how to improve the prediction, which is perhaps training an extra 10 more recordings, but we will proceed that in a later on iteration.

Then we removed all our present data and played with something new, instead of shapes, I wanted to see if the computer could recognize alphabets that I’m trying to spell out. Well, first I thought of the YMCA dance, because it’s an extremely “gestural” dance, but i figured just doing the dance the way it is isn’t so “rich” in movement, and I was certain the program wouldn’t be able to decode what alphabet I’m gesturing just by “pointing” in different directions (when you observe the YMCA dance, you’re angling your arms a certain way for each alphabet to spell it out, not so rich in movement…).

So I decided to write the alphabets out (like, literally WRITE it), and see what the computer predicts because it would be pretty cool to spell things on the computer just by using gestures etc. Unfortunately it failed miserably as practically none of the right alphabets were predicted. Again, we didn’t really delve into why this is happening other than acknowledging that the code is obviously not perfect and cannot be 100% accurate.

Reflection:

Machine learning, a subset of AI, we can train a certain code with enough data for it to be able to predict specific movements without explicit instructions. But what does this all mean? After the tinkering we’ve been doing, it’s time to step back and ideate what gestures we find particularly interesting and analyzing that, finding design opportunities perhaps, and how we can utilize this machine learning technology to a greater extent.

This all sounds exciting but I guess the limitation of the module is the phone itself. At the end of the day, the phone is quite chunky and it’s one single object, we cannot attach it on multiple joints for it to recognize “richer” gestures, we cannot attach it all five of our fingers so it knows how many fingers are being held up. Therefore, as a prediction, I feel like this module could potentially be quite challenging. I guess we shouldn’t be thinking about implementation just yet and focus on finding an interesting gesture as part of an activity to kick-off.