[Bug]: Jetson Nano Slow Performance With GPU/CUDA

[marcjasner]

What Operating System(s) are you seeing this problem on?

Other (plase, specify in the Steps to Reproduce)

dlib version

19.24

Python version

3.6

Compiler

gcc 7.5

Expected Behavior

I am attempting to create a facial detection/recognition component for a system I'm working on and I am unable to get dlib/face_recognition to perform at better than 2FPS under any circumstances.

The systems is running on a Jetson Nano 4G running Ubuntu 18.04 with Jetpack 4.6 installed.

I built dlib from scratch (using this helper script: https://github.com/JpnTr/Jetson-Nano-Install-Dlib-Library) and verified that suggested Jetson specific patches were made (as per https://medium.com/@ageitgey/build-a-hardware-based-face-recognition-system-for-150-with-the-nvidia-jetson-nano-and-python-a25cb8c891fd).

The test code I am running (against a single picture at 585x388 resolution with 5 people in it) looks like:

`#!/usr/bin/python3.6

import face_recognition
import time

def current_milli_time():
return round(time.time() * 1000)

for i in range(0,30):
t1=current_milli_time()
image = face_recognition.load_image_file("humans_1.jpg")
t2=current_milli_time()
face_locations = face_recognition.face_locations(image, model="cnn")
t3=current_milli_time()

print(face_locations)
print("load: ", t2-t1 )
print("detect: ",t3-t2)
print("Total: ", t3-t1)`

With no model specified (so the CPU is being used I believe) the normal face detection time is about 500ms, give or take. When I specify model="cnn" that number actually INCREASES to over 800ms.

tegrastats verifies that my GPU utilization is 99%.

I've seen this issue reported by other people but I have yet to see a solution. Shouldn't this be a reasonably fast operation (under 100ms) on a GPU? I've seen other (c/c++ based) face detection methods that suggest that detection can take as little as 20-50ms.

Current Behavior

Current behavior is that face detection takes 500ms on the CPU and even longer (800+ms) when using CUDA/GPU.

Steps to Reproduce

Nothing fancy, just run the code I provided.

Anything else?

No response

[davisking]

I don't know what's going on in face_recognition so I can't say (it's not a dlib package). Maybe there is a surprise in there.

[marcjasner]

Is there any logging I could turn on that would help me determine if the slowdown is from face_recognition or dlib?

What kind of timings SHOULD I be expecting? Is it safe to say 500-800ms is way too slow for GPU?

[davisking]

It depends on your GPU, how big the image is, and what kinds of settings or way of running it is being used in that code. So I can't say. Could be anything.

[marcjasner]

Ok, I appreciate the help. I opened a ticket with the face_recognition project once before and never got a reply. I'll give it another try. Thanks.

[marcjasner]

After some debugging of the face_recognition code I've traced through it enough to see that the code is performing quickly up until the call to dlib.face_recognition_model_v1(face_recognition_model), where face_recognition_model points to the file dlib_face_recognition_resnet_model_v1.dat

That call takes over 800ms to return on a Jetson Nano. Should I be calling a different function/model?

[arrufat]

Are you sure that model is running on CUDA? I am not familiar with the Jetson Nano speeds, but those timings you get look a lot like CPU inference. It should be really fast, since it's a slimmed version of ResNet34 that operates on 150×150 images.

[marcjasner]

When I run tegrastats I see GPU usage at 99%, so as far as I can tell it's running on the GPU. I also made sure I compiled dlib with CUDA support.

Also, if I don't specify that model="cnn" then the whole operation on CPU takes 500ms.

[arrufat]

Hmm, can you run the inference on the network twice in a row and measure only the second time? Maybe your measurements include the allocation on the GPU.

[marcjasner]

I actually have my test program running the inference 30 times in a row in a loop. the first call takes a LONG time (16-20s) and then after that the time is consistently 840ms (give or take a millisecond or two).

[marcjasner]

Oh, you mean the GPU utilization? it stays at 99% while the 30 inference loop runs.

[arrufat]

Then I don't know what's going on.

For reference, I timed the inference of one image using the dlib C++ example dnn_face_recognition_ex on a 12th Gen Intel® Core™ i7-1260P × 16 CPU, and it takes about 50 ms.

[arrufat]

Another thing, are you certain it's that model that's causing the latency? Not the face detector? How big are your images?

[marcjasner]

The test image is 585x388. How do I determine if it's the face detector or not? The dlib call that I put timing measurements around was the call to dlib.face_recognition_model_v1(face_recognition_model). Does that do the detection and the recognition or did I misunderstand this and miss another dlib call somewhere? That call was the one that took 840ms.

[marcjasner]

No, you're right, I'm looking at the wrong function... dlib.cnn_face_detection_model_v1(cnn_face_detection_model) is the function that is taking 840ms. The model it is using is mmod_human_face_detector.dat