docling-serve v1.13.0 and upwards give `No module named 'flash_attn_2_cuda' error

[TomNes94]

I run docling-serve on a server with Nvidia GPU support. I use the docling-serve-cu128 image (but have tried docling-serve-cu130 too). Versions below v1.13.0 work fine. The v1.13.0/v1.13.1 versions of docling-serve throw:

import flash_attn_2_cuda as flash_attn_gpu
ModuleNotFoundError: No module named 'flash_attn_2_cuda'

when using the surya_ocr plugin. I see others have had this issue in the past: #272. It was supposedly fixed in #292.

Anything I can change to solve this?

[dosubot]

Hi @TomNes94! I'm Dosu and I’m helping the docling-serve team.

The flash_attn_2_cuda error was fixed in PR #292 (merged July 2025), which should be included in v1.13.0+. If you're still seeing this error, here are some things to try:

1. Pull the latest image - make sure you're using the official images from the GitHub Container Registry:

docker pull ghcr.io/docling-project/docling-serve-cu128:latest

2. Try disabling flash-attention as a workaround:

docker run -p 5001:5001 --gpus all \
  -e DOCLING_DEVICE=cuda \
  -e DOCLING_CUDA_USE_FLASH_ATTENTION2=false \
  ghcr.io/docling-project/docling-serve-cu128:latest

3. Manual fix - if neither works, you can replace the flash-attn wheel with a compatible prebuilt one:

# Start container with bash
docker run --gpus all -it --rm --entrypoint /bin/bash \
  -p 5001:5001 ghcr.io/docling-project/docling-serve-cu128:latest

# Replace flash-attn
pip uninstall flash_attn
pip install flash_attn-2.8.0+cu128torch2.7-cp312-cp312-linux_x86_64.whl

Prebuilt wheels are available at mjun0812/flash-attention-prebuild-wheels.

The error originates from the layout extraction pipeline (RTDetrForObjectDetection), not specifically from SuryaOCR. Could you share which exact image tag/digest you're using? That would help confirm if this is a regression in the released v1.13.x images.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[TomNes94]

Step 3: manual fix worked 👍