Use batch size larger than 1 in models with dynamic output

[fgraffitti-cyberhawk]

I'm trying to run the face detector 206 on batches of images larger than 1.

I'm using an approach based on/readapted from this guide.

Here is my code:

import numpy as np
import openvino as ov
from openvino.runtime import PartialShape
import cv2

core = ov.runtime.ie_api.Core()
detection_model_xml = "face-detection-0206.xml"
detection_model = core.read_model(model=detection_model_xml)
detection_input_layer = detection_model.input(0) 

# the model looks like:
# <Model: 'torch-jit-export'
# inputs[
# <ConstOutput: names[image] shape[1,3,640,640] type: f32>
# ]
# outputs[
# <ConstOutput: names[boxes] shape[..750,5] type: f32>,
# <ConstOutput: names[labels] shape[..750] type: i64>
# ]>

new_shape = PartialShape([2,3,640,640]) # trying batch = 2, but ideally I'd like to use batch = -1 to support any batch size
detection_model.reshape({detection_input_layer.any_name: new_shape})
detection_compiled_model = core.compile_model(model=detection_model, device_name="CPU")

# the compiled model looks like
#<CompiledModel:
# inputs[
# <ConstOutput: names[image] shape[2,3,640,640] type: f32>
# ]
# outputs[
# <ConstOutput: names[boxes] shape[..750,5] type: f32>,
# <ConstOutput: names[labels] shape[..750] type: i64>
# ]>

# to run the model:
output = detection_compiled_model(input_data)

If I change the batch size of the input_data (it can be batch=2 images if I use PartialShape([2,3,640,640]), or batch=any size if I use PartialShape([-1,3,640,640])), the model might take longer for larger batchs, but the output is always the same, and corresponds to the predictions for the first image.

I suspect this is because the output layers are dynamic (boxes: shape[..750,5] and labels:shape[..750]) and don't get reshaped according to the input batch size.
However, I just started using openvino a couple of days ago so I'm not sure of how to fix the problem and allow for larger batches.

Any suggestion?

[Wovchena]

Hi. Sorry, I missed this. Yes, face-detection-0206 isn't usable with batch. You can try other detectors, for example face-detection-0205

[fgraffitti-cyberhawk]

Hi. Sorry, I missed this. Yes, face-detection-0206 isn't usable with batch. You can try other detectors, for example face-detection-0205

@Wovchena thanks for your reply, it's unfortunate that the model is not usable with batch. Looks like face-detection-0205 also has the same problem, output layer is still dynamic..

[Wovchena]

Sorry, my bad. It should have been face-detection-0204

import numpy as np
import openvino as ov
import cv2

core = ov.runtime.ie_api.Core()
detection_model_xml = r"C:/Users/vzlobin/.cache/omz/intel/face-detection-0204/FP16/face-detection-0204.xml"
detection_model = core.read_model(model=detection_model_xml)
detection_input_layer = detection_model.input(0) 

new_shape = (-1, 3, 448, 448)
detection_model.reshape({detection_input_layer.any_name: new_shape})
detection_compiled_model = core.compile_model(model=detection_model, device_name="CPU")

# to run the model:
images = np.stack([
    cv2.resize(cv2.imread(r"C:/Users/vzlobin/OneDrive - Intel Corporation/a/test-data/coco128/images/train2017/000000000036.jpg"), (new_shape[3], new_shape[2])),
    cv2.resize(cv2.imread(r"C:/Users/vzlobin/OneDrive - Intel Corporation/a/test-data/coco128/images/train2017/000000000086.jpg"), (new_shape[3], new_shape[2]))
])
input_data = images.transpose(0, 3, 1, 2)
output = detection_compiled_model(input_data)
BLUE = (255, 0, 0)
for image_id, _, conf, x_min, y_min, x_max, y_max in output['detection_out'][0, 0]:
    if image_id < 0:
        break
    if conf < 0.5:
        continue
    cv2.rectangle(images[int(image_id)], [round(x_min * new_shape[3]), round(y_min * new_shape[2])], [round(x_max * new_shape[3]), round(y_max * new_shape[2])], BLUE, 2)
cv2.imshow('winname',  np.concatenate(images, 1))
cv2.waitKey(0)