05-face-modals

Face Models

Slides

Objectives

• Learn about TensorFlow Face Landmarks Detection model and how it works.
• Learn to work with ml5.faceMesh.
• Learn to work with 3D graphics in p5.js WEBGL mode.

Lecture Notes

Tools

• ml5.faceMesh Reference
  • ml5.faceMesh Source Code
  • TensorFlow.js Face Landmarks Detection Source Code
  • MediaPipe FaceMesh Model Card

Related Projects

• Customizable AR face masks - Made with TensorFlow.js by Samarth Gulati and Praveen Sinha.
• Pose Animator by Shan Huang.

Music

• LipSync by YouTube by Google PI & bit.studio.
• Eye Conductor by Andreas Refsgaard.
• Sampler by Use All Five & Google Creative Lab.
• Mouth-Controlled Synthesizer with FaceMesh by Jack B. Du. [ Live Demo ]

Code Examples

• Class example: https://editor.p5js.org/yining/sketches/HHZ0bxKcA
• https://editor.p5js.org/yining/sketches/FNjlzk9hw
• https://editor.p5js.org/yining/sketches/p0ZwhM4tu
• face mask: https://editor.p5js.org/yining/sketches/jGWhn2bv5 download the face image here: https://github.com/samarthgulati/ar-face-filters/blob/master/assets/jamini-mask.jpg
• Face lip and tonge emoji: https://editor.p5js.org/yining/sketches/qRlAhiHLG
• Particle Eye: https://editor.p5js.org/yining/sketches/hJut1k5su import this particle class: https://editor.p5js.org/p5/sketches/Simulate:_ParticleSystem

Examples

• FaceMesh - image texture
• FaceMesh - pixel colors
• FaceMesh - lip contours

Core Functionalities

• FaceMesh - Single Image
• FaceMesh - Face Keypoints
• FaceMesh - Face Bounding Box
• FaceMesh - Parts Keypoints
• FaceMesh - Parts Bounding Box
• FaceMesh - Shapes from Parts

Creative Applications

• FaceMesh - Emoji Face by Jack Du
• FaceMesh - Triangulation by Jack Du
• FaceMesh - Face Masks by Jack Du

Technical Blog Posts

• Face and hand tracking in the browser with MediaPipe and TensorFlow.js by Ann Yuan and Andrey Vakunov. [ Original Paper ]

Video Tutorials

• ml5.js FaceMesh Creative Sketch - video tutorial with Patt Vira.

Assignment 5

1. Using FaceMesh, following the examples above and the ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model. Try to create an interaction that is surprising or one that is inspired by the project you find.
2. Document your research, your response, and your p5.js sketch in a blog post. Add a link to the post and your p5.js sketch on the Assignment 5 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.