After a series of ideation starting from listing a couple of specific activities consisted of rich gestures and concrete movements, we started leaning towards sports— ball sports, in particular. The most gestural ball sport you can think of is undoubtedly baseball. From the third base coach observing the overall scene and letting the catcher know, to the catcher signaling to the pitcher what type of ball he should pitch, to the umpire doing umpire-things…and so on. Not to mention that sports spectators often times have endless amounts of gestures to cheer on their favorite team.
When speaking of baseball, what struck out immediately if of course the pitching/throwing movement. When pitching a baseball, in addition to simply throwing the ball towards the home plate, a lot of thought has to be put into each pitch in order to support throwing a variety of pitches. Depending on your hand position, wrist position and angle of your arm, each ball will have a slightly different velocity, trajectory, and overall movement. Pitching the right ball at the right moment because it confuses the batter in various ways, gets batters and baserunners out, which contributes to the overall succession of the game.
Following this quick ideation we immediately moved on to exploring and tinkering with the machine learning code. We first recorded gestures in the terminal for a normal fastball, curveball, screwball, and throws we invented such as cowboy (elbow up, rotating your arm twice before throwing the ball), and frisbee (throwing the ball like you throw a frisbee). We recorded each gesture20 times, training them, then predicting them to see if machine learning can differentiate these different pitches. On my end I tried the screwball (inverted curveball), the frisbee, and the cowboy, and because these gestures are vastly different, machine learning was able to predict these motions fairly well.
On the other hand, Felix trained more similar range of motion such as the curveball and fastball, and that confused the system a little more, machine learning would often confuse all of these gestures as either just the curveball or just the fastball. It’s true that the motion is rather similar, though one has more wrist angle manipulation to achieve the “spinning” and the other with a quicker pitching velocity. We speculate that maybe it’s essential to make sure that when recording, we have to be careful with emphasizing the “spin” as well as the quicker motion for the fastball (less coordinate points), purposely making both pitches extremely unique from one another.
It’s extremely refreshing to see how powerful a simple machine learning code can be, even when it’s as simple as predicting gestures. There’s always a satisfying gush or a “I’m impressed” feeling whenever the right gesture is predicted. I look forward to working more with this and we talked about changing the number of lines read in the training and prediction code and perhaps recording and training ten additional lines to see if the predictions can be improved.
Journal with me! 