feat(ml): ML on Rockchip NPUs

[yoni13]

Goals: ML on Rockchip NPUs.
Testing on board: #13243 (reply in thread)

TODO:

• It's works on my OrangePi 3B (RK3566).
• Build Docker images.
• Build models for more SOCs.
• Able to set thread numbers for each model type (e.g. visual -> 2 threads) via environment variables.
• NPU core masks for RK3576/3588
• Decide model path (immich-app/ViT-B-32__openai/textual/rk3566.rknn) .
• Export script that accepts CLI arguments for the Immich model name and SoC, then exports the model.
• Able to maximize NPU usage by using rknnpool.py
• Documentation.
• Write tests.
• Model on Huggingface.
• Make Docker images work out of the box (without downloading models manually).

Nice to have:

• Test on PC and other arm-based boards to ensure it doesn't break something.
• Rebase my commits (sorry for ugly commit messages).
• Test if it's working on RK3588 (I don't have one).
• Support more models.

#13243

[yoni13]

Docker launch command:

docker run --security-opt systempaths=unconfined --security-opt apparmor=unconfined --device /dev/dri --device /dev/dma_heap --device /dev/rga --device /dev/mpp_service -v /cache:/cache:ro  -p 3004:3003 -v /sys/kernel/debug/:/sys/kernel/debug/:ro --name rknnimmich_name -d rknnimmich

and it works (if you download model to cache ofc)

ViT-B-32 and buffalo_l are loaded with two threads rerunning jobs: 2.7G, and peak 3.5G RAM. (its like running 4 models at the same time)
Update: this statistic is before we only load onnx when required, will update mem usage when I got time

[yoni13]

Most issues mentioned in the review should be fixed now.

[yoni13]

Wrote a matrix build script to see what model is not supported, all of them are nlib and XLM.
https://github.com/yoni13/immich_to_rknn2/actions/runs/12892750773

[mertalev]

Wrote a matrix build script to see what model is not supported, all of them are nlib and XLM. yoni13/immich_to_rknn2/actions/runs/12892750773

Oh wow, that is pretty cool!

I have an RK3588, so I can do some testing with this PR soon.

[yoni13]

3000 photos, Everything default, rerun smart search, face detection&recognition
RK3566 NPU:

• 50mins
• 65 c

RK3566 CPU:

• 4hr 37min
• 76c

GPU?
/dev/mali0 on RK3566 doesn't even work on Armbian/DietPi

[mertalev]

Sorry, just a heads up that I will be away for the next two weeks and won't be able to test it locally or upload the converted models to Hugging Face until after I'm back.

From another quick look at the PR, the only thing that still stands out is the ugly shape handling, but I don't consider it a blocker. However, I'd like to hold off from merging this until I'm back so we can prepare the models and be available to handle potential bug fixes after release.

[NicholasFlamy]

I've got an 8GB Orange Pi 5 with a RK3588S (S has slightly less IO, otherwise identical). I may be able to test it sometime this week.

[yoni13]

Oh btw, I am not able to test RKNN's CORE auto mapping because I have a rk3566 which only has one core. Hope their auto mapping is useful, or maybe we have to manually setting cores by tasks. Nicholas Flamy ***@***.***> 於 2025年2月6日 週四 上午1:04 寫道：

…

I've got an 8GB Orange Pi 5 with a RK3588S (S has slightly less IO, otherwise identical). I may be able to test it sometime this week. — Reply to this email directly, view it on GitHub <#15241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASF5CKDRZOYCAXYM7NTZX4D2OJADVAVCNFSM6AAAAABU7QYIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZXGUYTQMZVHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>