Our focus is to enable a user to command Cozmo (an educational toy robot) using hand gestures. In particular, Cozmo will be able to turn to an area the user is pointing towards.
Using transfer learning, we taught a neural net how to distinguish pictures with hands from those without. Once Cozmo recognizes that the hand gesture is a point, he identifies where the fingertip is, and based on this, decides where to turn. To find the fingertip, we filter out the pink post-it that the user is wearing and calculate the centroid to turn in that direction.