04-hand-models

Hand Models

Slides

Objectives

• Learn about TensorFlow Hand Pose Detection model and how it works.
• Learn to work with ml5.handPose.
• Learn to work with 3D graphics in p5.js WEBGL mode.

Lecture Notes

Tools

• ml5.handPose Reference
  • ml5.handPose Souce Code
  • TensorFlow.js Hand Pose Detection Source Code
  • Hand Detection/Tracking Model Card

Related Projects

• Keyboard by Use All Five & Google Creative Lab.
• Fingerspelling by Hello Monday.
• Projection Mapping by Nahuel Gerth.

Music

• Melody Painter with HandPose by Jack B. Du. [ Live Demo ]

Playful

• Finger Talk by Future Sketches.
• Look At You by Alan Ren & Kyrie Yang.
• Bubbles by Nahuel Gerth.
• Finger Numerals by Nahuel Gerth.

Code Examples

In class examples

• HandPose - pinch painting
• Woooooow: https://editor.p5js.org/yining/sketches/uiG-8dfi8: use two index fingers to trigger "wooow"
• https://editor.p5js.org/yining/sketches/ND2zXhqdr (Thumb tip to Index finger tip)
• https://editor.p5js.org/yining/sketches/aN6giRBWb (Looooove)
• https://editor.p5js.org/yining/sketches/hP7oisufl (Heart Gesture)
• https://editor.p5js.org/yining/sketches/vQ2lYkYiW (Heart Gesture - Particle system)

Core Functionalities

• HandPose - Single Image
• HandPose - Keypoints
• HandPose - Keypoints 3D
• HandPose - Parts
• HandPose - Detect Start and Stop
• Training hand pose model
  • Hand pose tracking + KNN Classification, p5 web editor code, demo video
  • Hand pose tracking + Neural Network, demo video
    • Collect data
    • Train the model
    • Run the model

Creative Applications

• HandPose - Particles by Jack Du
• HandPose - Quadrilateral by Jack Du
• HandPose - Quadrilateral with Texture by Jack Du

Technical Blog Posts

• On-Device, Real-Time Hand Tracking with MediaPipe by Valentin Bazarevsky and Fan Zhang.
• 3D Hand Pose with MediaPipe and TensorFlow.js by Valentin Bazarevsky, Ivan Grishchenko, Eduard Gabriel Bazavan, Andrei Zanfir, Mihai Zanfir, Jiuqiang Tang, Jason Mayes, Ahmed Sabie.

Video Tutorials

Assignment 4

1. Research and find a project (experiments, websites, art installations, games, etc) that utilizes machine learning in a creative way. Consider the following:
   • What type of machine learning models did the creator use?
   • What data might have been used to train the machine learning model?
   • Why did the creator of the project choose to use this machine learning model?
2. Using HandPose, following the examples above and the ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model. Try to create an interaction that is surprising or one that is inspired by the project you find.
3. Document your research, your response, and your p5.js sketch in a blog post. Add a link to the post and your p5.js sketch on the Assignment 4 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.