integrate Apple ML Depth Pro backend with automated model downloading

[saribx]

Requesting Review

@Tenebrae311 @fhilgers @Paul2607 @ChristophMantsch @BenediktZinn @anuunchin @zohrehasadi00

• Integrated Apple ML Depth Pro as an additional (more accurate) option
• It provides metric depth (actual distances in meters) instead of relative depth
• Because of the large file size I couldn't test it (running on mobile data)
• 'Make' commands added for easy installation and usage (as well as automatic donwload, if not already, when trying to use the model with the analyser)
• See the documentation for a overview: https://docs.google.com/document/d/1_ViqPQN2cYG1juIJH8HXBN0SNQAlGB_6vvfPBfwFfuU/edit?tab=t.0

[zohrehasadi00]

It generally looked promising, code was making sense, didn't look too deep to it tho!!

[fhilgers]

@zohrehasadi00 were you able to verify that it works? Apple depth pro seems to crash my laptop or make it freeze

[fhilgers]

Apart from that I cannot actually infer with the model as my computer is not capable enough:

• unexpected keyword argument 'aspect_ratio'

❯ DEPTH_BACKEND=depth_pro make run-analyzer-local
Starting analyzer service on port 8001...
cd src/backend && uv run uvicorn analyzer.main:app --host 0.0.0.0 --port 8001 --reload
INFO:     Will watch for changes in these directories: ['/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend']
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
INFO:     Started reloader process [74135] using StatReload
INFO:     Started server process [74137]
INFO:     Waiting for application startup.
{"timestamp": "2026-01-13T21:54:29.778503+00:00", "level": "INFO", "logger": "common.core.detector", "message": "Initializing Object Detector with backend: torch", "service": "analyzer", "environment": "development"}
Downloading https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.pt to '/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/models/yolo11n.pt'...
100%|██████████████████████████████████████████████████████████████████████████████| 5.35M/5.35M [00:00<00:00, 24.9MB/s]
{"timestamp": "2026-01-13T21:54:30.786488+00:00", "level": "INFO", "logger": "common.core.depth", "message": "Initializing Depth Estimator with backend: depth_pro", "service": "analyzer", "environment": "development"}
{"timestamp": "2026-01-13T21:54:30.786709+00:00", "level": "INFO", "logger": "common.core.depth_pro", "message": "Loading Depth Pro model...", "service": "analyzer", "environment": "development"}
{"timestamp": "2026-01-13T21:54:30.786767+00:00", "level": "ERROR", "logger": "common.core.depth_pro", "message": "Failed to initialize Depth Pro: create_model_and_transforms() got an unexpected keyword argument 'aspect_ratio'", "service": "analyzer", "environment": "development"}
ERROR:    Traceback (most recent call last):
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/home/fhilgers/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/home/fhilgers/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/analyzer/main.py", line 55, in lifespan_context
    get_depth_estimator(midas_cache_directory)
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth.py", line 75, in get_depth_estimator
    _depth_estimator = _depth_estimator_factory(midas_cache_directory)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth.py", line 58, in _default_depth_estimator_factory
    return factory(midas_cache_directory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth_pro.py", line 50, in __init__
    self.model, self.transform = depth_pro.create_model_and_transforms(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: create_model_and_transforms() got an unexpected keyword argument 'aspect_ratio'

ERROR:    Application startup failed. Exiting.

• The model is not downloaded automatically when doing DEPTH_BACKEND=depth_pro make run-analyzer-local

ERROR:    Traceback (most recent call last):
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/home/fhilgers/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/home/fhilgers/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/analyzer/main.py", line 55, in lifespan_context
    get_depth_estimator(midas_cache_directory)
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth.py", line 75, in get_depth_estimator
    _depth_estimator = _depth_estimator_factory(midas_cache_directory)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth.py", line 58, in _default_depth_estimator_factory
    return factory(midas_cache_directory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/common/core/depth_pro.py", line 50, in __init__
    self.model, self.transform = depth_pro.create_model_and_transforms(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/depth_pro/depth_pro.py", line 135, in create_model_and_transforms
    state_dict = torch.load(config.checkpoint_uri, map_location="cpu")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/ultralytics/utils/patches.py", line 86, in torch_load
    return _torch_load(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/torch/serialization.py", line 1484, in load
    with _open_file_like(f, "rb") as opened_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/torch/serialization.py", line 759, in _open_file_like
    return _open_file(name_or_buffer, mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.B5Zm4JGgC7/amos2025ws04-robot-visual-perception/src/backend/.venv/lib/python3.11/site-packages/torch/serialization.py", line 740, in __init__
    super().__init__(open(name, mode))
                     ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/depth_pro.pt'

• when downloading, uninstalls the packages that are installed per default via make install:

❯ make download-depth-pro
Downloading Depth Pro model...
cd src/backend && uv sync --extra inference
Resolved 133 packages in 1ms
Uninstalled 25 packages in 236ms
 - attrs==25.4.0
 - binaryornot==0.4.4
 - boolean-py==5.0
 - chardet==5.2.0
 - colorama==0.4.6
 - coloredlogs==15.0.1
 - flatbuffers==25.9.23
 - humanfriendly==10.0
 - iniconfig==2.3.0
 - license-expression==30.4.4
 - ml-dtypes==0.5.4
 - mypy==1.13.0
 - mypy-extensions==1.1.0
 - onnx==1.19.1
 - onnx-ir==0.1.12
 - onnxruntime==1.20.1
 - onnxscript==0.5.6
 - onnxslim==0.1.75
 - pluggy==1.6.0
 - pytest==8.3.3
 - pytest-asyncio==1.2.0
 - python-debian==1.0.1
 - reuse==4.0.3
 - ruff==0.7.0
 - tomlkit==0.13.3
cd src/backend && uv run python ../../scripts/download_models.py \
  --models depth-pro \
  --output-dir models

• self.model_type is not set on the DepthProEstimator

I don't know whether it is worth it to fix it at the moment when there is nobody that can actually verify if anything really works. @BenediktZinn and I just tried it on his nvidia GPU and even there it goes out of memory without any detection being shown...

[fhilgers]

@saribx is there an update on this?

[fhilgers]

@saribx ping