Failed to run background_subtraction_demo_gapi with Media Files Available for Demos

[kminemur]

Hi team, background_subtraction_demo_gapi failed with person-bicycle-car-detection.mp4 from Media Files: https://storage.openvinotoolkit.org/data/test_data/videos.

How to download:
wget https://storage.openvinotoolkit.org/data/test_data/videos/person-bicycle-car-detection.mp4

[Environment]
OS: Ubuntu 22.04.2 LTS
Kernel: 5.15.0-86-generic
Openvino: 2023.3.0
OpenCV: 4.9.0

[Log]
$ ./background_subtraction_demo_gapi -m $HOME/workshop/kazuki/open_model_zoo/demos/background_subtraction_demo/cpp_gapi/intel/instance-segmentation-security-0002/FP32/instance-segmentation-security-0002.xml -i person-bicycle-car-detection.mp4 -at maskrcnn
[ INFO ] OpenVINO
[ INFO ] version: 2023.3.0
[ INFO ] build: 2023.3.0-13775-ceeafaf64f3
[ INFO ] The background matting model /home/yuzhang3/workshop/kazuki/open_model_zoo/demos/background_subtraction_demo/cpp_gapi/intel/instance-segmentation-security-0002/FP32/instance-segmentation-security-0002.xml is loaded to CPU device.
[ ERROR ] [ GENERAL_ERROR ] Exception from src/plugins/intel_cpu/src/infer_request.cpp:393:
Can't set the input tensor with name: image, because the model input (shape=[1,3,?,?]) and the tensor (shape=(1.432.768.3)) are incompatible

[Siddharth-Latthe-07]

@Wovchena I guess, this issue occurs due to mismatch between the expected input shape of the model and the actual input shape provided by the background_subtraction_demo_gapi demo.
you may try these steps and do let me know if it works:-

1. resize the video frame to the required dimensions before passing it to the model.
2. snippet for preprocessing video frames and convert to required format of [1, 3, height, width].

import cv2
import numpy as np

def preprocess_frame(frame, target_height, target_width):
    # Resize the frame
    resized_frame = cv2.resize(frame, (target_width, target_height))
    # Convert the frame to the format [1, 3, height, width]
    input_frame = np.expand_dims(resized_frame.transpose(2, 0, 1), axis=0)
    return input_frame

# Open the video file
cap = cv2.VideoCapture('person-bicycle-car-detection.mp4')

# Read the first frame
ret, frame = cap.read()
if not ret:
    print("Failed to read the video file")
    exit()

# Preprocess the frame
input_frame = preprocess_frame(frame, 432, 768)  # Replace with the target height and width

# Save the preprocessed frame as an example (optional)
np.save('preprocessed_frame.npy', input_frame)

# Release the video capture object
cap.release()

3. update the demo script with the above change and execute it

plz let me know, if the above works or not
Thanks

[Wovchena]

The problem is in GAPI which is a part of OpenCV. @TolyaTalamanov and @DariaMityagina were working on fixing. Apparently they dropped this task.

[TolyaTalamanov]

@Siddharth-Latthe-07 The problem occurs because of dynamic nature of the model.

Could you try to fix model input to "some" static shape and re-run demo, please?
https://docs.openvino.ai/2024/openvino-workflow/running-inference/dynamic-shapes.html

[Siddharth-Latthe-07]

@TolyaTalamanov , Here's my approach to address the above issue sent by you (I tried setting a static input shape for the model using OpenVINO's tools or APIs that allow input resizing.) :-

Step 1: Set the Model Input Shape Statically
explicitly set the input shape to a fixed value. can be done by using the OpenVINO Model Optimizer or directly setting it in code.
(mo.py --input_model instance-segmentation-security-0002.xml --input_shape [1,3,432,768]) OR

import openvino.runtime as ov

# Load the model
core = ov.Core()
model = core.read_model(model="instance-segmentation-security-0002.xml")
compiled_model = core.compile_model(model, device_name="CPU")

# Set a static input shape
input_tensor = compiled_model.input(0)
model.reshape({input_tensor.any_name: [1, 3, 432, 768]})

# Load the new reshaped model
compiled_model = core.compile_model(model, "CPU")

This sets the input to a fixed shape before inference.

Step 2: Update the Demo Script
input_frame = preprocess_frame(frame, 432, 768) (Should match with fixed model input)

After making these changes, re-run the background_subtraction_demo_gapi. This should eliminate the GENERAL_ERROR related to mismatched input shapes.

[TolyaTalamanov]

@Siddharth-Latthe-07 You don't need to update the demo itself. Suggestion is to eliminate dynamism from the model by reshaping it to static shape

[Siddharth-Latthe-07]

@TolyaTalamanov, got it

1. Reshape the model:-
   Use the OpenVINO Python API to reshape the model to a static input shape:

from openvino.runtime import Core

# Initialize the OpenVINO runtime
core = Core()

# Load the model
model = core.read_model(model="instance-segmentation-security-0002.xml")

2. Reshape the Model to Static Input Dimensions:
   Set the model's input shape explicitly to a static value.
   model.reshape({"image": [1, 3, 432, 768]})

3. Compile the Model for Inference:
   After reshaping, compile the model for the target device (e.g., CPU):
   compiled_model = core.compile_model(model, device_name="CPU")

4. Update the OpenVINO Model File
   Once you have reshaped the model using the OpenVINO API, save the updated model to use with your demo:

from openvino.runtime import serialize

serialize(model, "instance-segmentation-security-0002_reshaped.xml", "instance-segmentation-security-0002_reshaped.bin")

5. Run the Demo with the Updated Model
   Use the reshaped model XML file (instance-segmentation-security-0002_reshaped.xml) for running the background_subtraction_demo_gapi:
   ./background_subtraction_demo_gapi -m path/to/instance-segmentation-security-0002_reshaped.xml -i person-bicycle-car-detection.mp4 -at maskrcnn

By reshaping the model to a static input shape, you eliminate the dynamic nature of the model, which should resolve the input shape compatibility issue during inference.