The system uses an A.I. model to analyze the video feed from the web camera and predict which hand pose is in the video. The robot controller gets this information and commands the robot’s axes motion. As the robot is completely independent from the A.I. model, they communicate through a wireless interface.

The architecture of the model is based on the ResNet50 architecture for image classification. We used transfer learning and fine-tuning methods to apply it to hand pose recognition. The model is made of more than 32 million parameters and can be run on a computer with a GPU (GeForce GTX 1080 Ti) with a response time less than 20 milliseconds which shows real-time control capacity of the robot.

We have also changed the basic architecture to a smaller one (ten times smaller) and then managed to run the system on a much smaller (and much cheaper, ~30$) computer, a Raspberry Pi 3. As a Raspberry Pi does not have the same computation capability as a GPU, the processing time is affected (from 20 milliseconds to 1 second).