RAG mode is using cuda in a ROCm system

[jalberto]

Issue Description

when running ramalama in fedora silverblue on AMD (strix halo) it tries to use CUDA and crash

Steps to reproduce the issue

ramalama rag ./foo.pdf local/myrag

➜ ramalama rag ./foo.pdf local/myrag
Converting foo.pdf .   2025-10-31 12:24:24,154 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]
2025-10-31 12:24:24,162 - INFO - Going to convert document batch...
2025-10-31 12:24:24,162 - INFO - Initializing pipeline for StandardPdfPipeline with options hash 75463f421d05cb4304e1f714cf00d35d
2025-10-31 12:24:24,166 - INFO - Loading plugin 'docling_defaults'
2025-10-31 12:24:24,166 - INFO - Registered picture descriptions: ['vlm', 'api']
2025-10-31 12:24:24,170 - INFO - Loading plugin 'docling_defaults'
2025-10-31 12:24:24,170 - INFO - Registered ocr engines: ['auto', 'easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
Converting foo.pdf ..   2025-10-31 12:24:27,612 - INFO - Accelerator device: 'cuda:0'
Converting foo.pdf ...   2025-10-31 12:24:41,667 - INFO - Accelerator device: 'cuda:0'
Converting foo.pdf .   2025-10-31 12:24:42,157 - INFO - Processing document foo.pdf
2025-10-31 12:24:42,317 - WARNING - Encountered an error during conversion of document c33927cdfcd4b22717a49932168d476cda8d350d6348494cff58ac69822272b7:
Traceback (most recent call last)

Describe the results you received

it crash with a python stack trace

Describe the results you expected

to use ROCm to access the GPU for processing

ramalama info output

"mistral": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral-small3.1": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral-small3.1:24b": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral:7b": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
            "mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
            "mistral:7b-v3": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
            "mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
            "mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
            "openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
            "openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
            "phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
            "qwen2.5vl": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:2b": "hf://ggml-org/Qwen2.5-VL-2B-Instruct-GGUF",
            "qwen2.5vl:32b": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:3b": "hf://ggml-org/Qwen2.5-VL-3B-Instruct-GGUF",
            "qwen2.5vl:7b": "hf://ggml-org/Qwen2.5-VL-7B-Instruct-GGUF",
            "smollm:135m": "hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF",
            "smolvlm": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "smolvlm:256m": "hf://ggml-org/SmolVLM-256M-Instruct-GGUF",
            "smolvlm:2b": "hf://ggml-org/SmolVLM-Instruct-GGUF",
            "smolvlm:500m": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "stories-be:260k": "hf://taronaeo/tinyllamas-BE/stories260K-be.gguf",
            "tiny": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
            "tinyllama": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
        }
    },
    "Store": "/var/home/vl/.local/share/ramalama",
    "UseContainer": true,
    "Version": "0.13.0"
}

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[rhatdan]

@mikebonnet PTAL

[jalberto]

maybe related: docling-project/docling-serve#427

[mikebonnet]

We've made significant changes to the way rag operations work in the latest release. Please retry with 0.14.0. If the issue persists, please provide the output using ramalama --debug.

[jalberto]

I got different error now:

Trying to pull quay.io/ramalama/rocm-rag:latest...
Getting image source signatures
Copying blob cacf8668606b done   |
Copying blob 3920358cdde5 done   |
Copying config 35322dffbf done   |
Writing manifest to image destination
Converting foo.pdf ..   2025-11-03 09:49:49,919 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:49,949 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:49,979 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

len(pages)=1, 0-0
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 1-1
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 2-2
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 3-3
len(valid_pages)=1
len(valid_page_images)=1
Converting foo.pdf ...   2025-11-03 09:49:50,012 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,040 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,068 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,095 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,118 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,148 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,177 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:49:50,198 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Error: Conversion failed for: foo.pdf with status: ConversionStatus.FAILURE
len(pages)=1, 4-4
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 5-5
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 6-6
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 7-7
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 8-8
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 9-9
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 10-10
len(valid_pages)=1
len(valid_page_images)=1
Error: Command '['podman', 'run', '--rm', '--label', 'ai.ramalama.engine=podman', '--label', 'ai.ramalama.runtime=llama.cpp', '--label', 'ai.ramalama.command=rag', '--security-opt=label=disable', '--pull', 'newer', '--device', '/dev/dri', '--device', '/dev/kfd', '--device', '/dev/accel', '-e', 'HIP_VISIBLE_DEVICES=0', '-v', '/var/home/vl/foo.pdf:/docs/foo.pdf:ro', '-v', '/tmp/RamaLama_rag_0209atfr/vectordb:/output:rw', 'quay.io/ramalama/rocm-rag:latest', 'doc2rag', '--format', 'qdrant', '/output', '/docs']' returned non-zero exit status 1.

Debug:

 ramalama --debug rag ./foo.pdf local/myrag
2025-11-03 10:58:25 - DEBUG - run_cmd: podman run --rm --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.command=rag --security-opt=label=disable --pull newer --device /dev/dri --device /dev/kfd --device /dev/accel -e HIP_VISIBLE_DEVICES=0 -v /var/home/vl/foo.pdf:/docs/foo.pdf:ro -v /tmp/RamaLama_rag_6ze6lrkw/vectordb:/output:rw quay.io/ramalama/rocm-rag:latest doc2rag --debug --format qdrant /output /docs
DEBUG:ramalama:run_cmd: podman run --rm --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.command=rag --security-opt=label=disable --pull newer --device /dev/dri --device /dev/kfd --device /dev/accel -e HIP_VISIBLE_DEVICES=0 -v /var/home/vl/foo.pdf:/docs/foo.pdf:ro -v /tmp/RamaLama_rag_6ze6lrkw/vectordb:/output:rw quay.io/ramalama/rocm-rag:latest doc2rag --debug --format qdrant /output /docs
2025-11-03 10:58:25 - DEBUG - Working directory: None
DEBUG:ramalama:Working directory: None
2025-11-03 10:58:25 - DEBUG - Ignore stderr: False
DEBUG:ramalama:Ignore stderr: False
2025-11-03 10:58:25 - DEBUG - Ignore all: False
DEBUG:ramalama:Ignore all: False
2025-11-03 10:58:25 - DEBUG - env: None
DEBUG:ramalama:env: None
Converting foo.pdf .   2025-11-03 09:58:29,900 docling.datamodel.document INFO: detected formats: [<InputFormat.PDF: 'pdf'>]
2025-11-03 09:58:29,908 docling.document_converter INFO: Going to convert document batch...
2025-11-03 09:58:29,908 docling.document_converter INFO: Initializing pipeline for StandardPdfPipeline with options hash 4a89bda5a26a77cb2b7befc50a30d131
2025-11-03 09:58:29,913 docling.models.factories.base_factory INFO: Loading plugin 'docling_defaults'
2025-11-03 09:58:29,914 docling.models.factories INFO: Registered picture descriptions: ['vlm', 'api']
2025-11-03 09:58:29,918 docling.models.factories.base_factory INFO: Loading plugin 'docling_defaults'
2025-11-03 09:58:29,919 docling.models.factories INFO: Registered ocr engines: ['auto', 'easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
Converting foo.pdf ..   2025-11-03 09:58:32,925 docling.utils.accelerator_utils INFO: Accelerator device: 'cuda:0'
2025-11-03 09:58:32,927 urllib3.connectionpool DEBUG: Starting new HTTPS connection (1): huggingface.co:443
2025-11-03 09:58:33,171 urllib3.connectionpool DEBUG: https://huggingface.co:443 "GET /api/models/ds4sd/docling-layout-heron/revision/main HTTP/1.1" 307 97
2025-11-03 09:58:33,402 urllib3.connectionpool DEBUG: https://huggingface.co:443 "GET /api/models/docling-project/docling-layout-heron/revision/main HTTP/1.1" 200 912
Converting foo.pdf .   2025-11-03 09:58:33,979 docling_ibm_models.layoutmodel.layout_predictor DEBUG: LayoutPredictor settings: {'model_name': 'RTDetrV2ForObjectDetection', 'safe_tensors_file': '/var/cache/huggingface/hub/models--ds4sd--docling-layout-heron/snapshots/bdb7099d742220552d703932cc0ce0a26a7a8da8/model.safetensors', 'device': 'cuda', 'num_threads': 4, 'image_size': {'height': 640, 'width': 640}, 'threshold': 0.3}
2025-11-03 09:58:34,317 urllib3.connectionpool DEBUG: https://huggingface.co:443 "GET /api/models/ds4sd/docling-models/revision/v2.3.0 HTTP/1.1" 307 93
Converting foo.pdf ..   2025-11-03 09:58:37,747 urllib3.connectionpool DEBUG: https://huggingface.co:443 "GET /api/models/docling-project/docling-models/revision/v2.3.0 HTTP/1.1" 200 1110
2025-11-03 09:58:37,769 docling.utils.accelerator_utils INFO: Accelerator device: 'cuda:0'
Converting foo.pdf ...   2025-11-03 09:58:38,193 docling.pipeline.base_pipeline INFO: Processing document foo.pdf
2025-11-03 09:58:38,237 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,266 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,296 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,328 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,354 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,382 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,408 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

len(pages)=1, 0-0
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 1-1
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 2-2
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 3-3
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 4-4
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 5-5
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 6-6
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 7-7
len(valid_pages)=1
len(valid_page_images)=1
Converting foo.pdf .   2025-11-03 09:58:38,432 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,461 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,489 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,510 docling.pipeline.standard_pdf_pipeline ERROR: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

2025-11-03 09:58:38,513 docling.document_converter INFO: Finished converting document foo.pdf in 8.61 sec.
Error: Conversion failed for: foo.pdf with status: ConversionStatus.FAILURE
len(pages)=1, 8-8
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 9-9
len(valid_pages)=1
len(valid_page_images)=1
len(pages)=1, 10-10
len(valid_pages)=1
len(valid_page_images)=1
Error: Command '['podman', 'run', '--rm', '--label', 'ai.ramalama.engine=podman', '--label', 'ai.ramalama.runtime=llama.cpp', '--label', 'ai.ramalama.command=rag', '--security-opt=label=disable', '--pull', 'newer', '--device', '/dev/dri', '--device', '/dev/kfd', '--device', '/dev/accel', '-e', 'HIP_VISIBLE_DEVICES=0', '-v', '/var/home/vl/foo.pdf:/docs/foo.pdf:ro', '-v', '/tmp/RamaLama_rag_6ze6lrkw/vectordb:/output:rw', 'quay.io/ramalama/rocm-rag:latest', 'doc2rag', '--debug', '--format', 'qdrant', '/output', '/docs']' returned non-zero exit status 1.