Skip to content

FaceLandmarker is non-deterministic in VIDEO/LIVE-STREAM mode #5253

Open
@twoertwein

Description

@twoertwein

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Mac M2

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

python

MediaPipe version

0.10.9 (latest version working on Mac - #5168)

Bazel version

No response

Solution

FaceLandmarker

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

The same image should produce the same results.

Describe the expected behaviour

Consecutive calls to detect_for_video return different results for the same image.

Standalone code/steps you may have used to try to get what you need

from mediapipe import Image
from mediapipe.tasks.python import BaseOptions
from mediapipe.tasks.python.vision import (
    FaceLandmarker,
    FaceLandmarkerOptions,
    RunningMode,
)

base_options = BaseOptions(
    model_asset_path="face_landmarker_v2_with_blendshapes.task"
)
options = FaceLandmarkerOptions(
    base_options=base_options,
    running_mode=RunningMode.VIDEO, # LIVE_STREAM has the same issue as video
    output_face_blendshapes=True,
    num_faces=1,
)

image = Image.create_from_file("neutral_face_photo.png")

with FaceLandmarker.create_from_options(options) as model:
    for timestamp in range(10):
        first_blendshape = model.detect_for_video(image, timestamp).face_blendshapes[0][0].score
        print(timestamp, first_blendshape)
        # detect / IMAGE: 3.8817524909973145e-06

Other info / Complete Logs

0 3.8817524909973145e-06
1 3.895277131960029e-06
2 3.8954626688791905e-06
3 3.896072030329378e-06
4 3.8940884223848116e-06
5 3.894496785505908e-06
6 3.8950020098127425e-06
7 3.893160283041652e-06
8 3.893947450706037e-06
9 3.89325668948004e-06

Metadata

Metadata

Assignees

Labels

os:macOSIssues on MacOSplatform:pythonMediaPipe Python issuestask:face landmarkerIssues related to Face Landmarker: Identify facial features for visual effects and avatars.type:bugBug in the Source Code of MediaPipe Solution

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions