Picture descriptions missing from export_to_markdown() until after JSON round-trip

[lakatosd]

Bug

When converting a PDF with do_picture_description=True, the initial result.document.export_to_markdown() does not include the generated image descriptions. However, after saving the document to JSON and loading it back (DoclingDocument.load_from_json), export_to_markdown() does include the descriptions.

Observations:

• In the initial result.document, doc.pictures entries have meta=None and only annotations.
• After JSON round-trip, loaded_doc.pictures contain both annotations and meta, and the markdown includes the descriptions.
• Expected behavior: both the initial document and the loaded one should include the generated picture descriptions in markdown.
• It looks like the picture description metadata is generated but not attached (or not surfaced) to doc.pictures[*].meta before serialization. The JSON round-trip likely triggers normalization that merges annotations into meta, enabling markdown rendering to pick them up.

Steps to reproduce

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions, PictureDescriptionApiOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling_core.types import DoclingDocument
import os


def openai_compatible_vlm_options(model: str, prompt: str, max_tokens: int = 4096, api_key: str = ""):
    headers = {}
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"

    options = PictureDescriptionApiOptions(
        url="https://api.openai.com/v1/chat/completions",
        params=dict(
            model=model,
            max_tokens=max_tokens,
        ),
        headers=headers,
        prompt=prompt,
        timeout=90,
        scale=2.0,
    )
    return options

converter = DocumentConverter()
pipeline_options = PdfPipelineOptions()
pipeline_options.enable_remote_services = True
pipeline_options.do_picture_description = True
pipeline_options.picture_description_options = openai_compatible_vlm_options(
    model="gpt-4.1-mini",
    prompt="Describe the image in three sentences. Be consise and accurate.",
    api_key=os.getenv("OPENAI_API_KEY"),
    )
converter.format_to_options[InputFormat.PDF] = PdfFormatOption(pipeline_options=pipeline_options)


source = "https://arxiv.org/pdf/2408.09869"
result = converter.convert(source)

markdown = result.document.export_to_markdown()  # does not contain the image description

result.document.save_as_json("docling_test.json")
loaded_doc = DoclingDocument.load_from_json("docling_test.json")
loaded_markdown = loaded_doc.export_to_markdown() # contains the image description

Docling version

2.61.2

Python version

Python 3.10.19

[dosubot]

Hi @lakatosd! I'm Dosu and I’m helping the docling team.

This happens because picture descriptions are initially attached as annotations on each PictureItem, but the markdown export expects them in the meta field. The JSON round-trip (save and reload) triggers a model validator that migrates annotations into meta, making the descriptions available for markdown export. That's why you only see the descriptions after reloading from JSON—before that, meta is None and export_to_markdown skips the descriptions details.

There isn't a public function to trigger this migration in-memory, but you can force it by re-instantiating the document from its dict (which runs model validation):

from docling_core.types import DoclingDocument

# After conversion
doc = result.document
doc = DoclingDocument.model_validate(doc.model_dump())
markdown = doc.export_to_markdown()

This will migrate annotations into meta without needing to write/read JSON. If you want a more robust or custom solution, you could patch the serializer or write a utility to migrate annotations to meta directly example workaround.

This behavior is known and discussed in the project; see issue #2560 and issue #1226 for more background.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[mmb78]

I see the same problem. Your workaround is interesting! I just tested this using my script and it works!
I think this behavior (no image descriptions in MD) should be fixed! Unfortunately, the suggestion from the Dosu bot are not helpful and the team clearly sees this as a feature and not a bug (read here)

[lakatosd]

@dosu Thanks for the detailed explanation and the workaround.

This is not expected behavior on our side: export_to_markdown() should produce the same output for the freshly converted document as it does after a JSON round-trip. Having markdown export depend on whether the document has been serialized/reloaded is a bug.

[dosubot]

You're right—having export_to_markdown() output depend on whether the document has been serialized or reloaded is inconsistent and not ideal. Technically, this happens because picture descriptions are first attached as annotations, but the markdown export expects them in the meta field. The migration from annotations to meta only runs during (de)serialization, which is why you see the difference after a JSON round-trip.

The maintainers currently treat this as a feature, not a bug, due to the design of the markdown serializer and the lossy nature of markdown export. There’s no open issue or pull request indicating a planned fix to make export_to_markdown consistent in this regard. The recommended workaround is to re-instantiate the document with model validation in-memory (e.g., DoclingDocument.model_validate(doc.model_dump())) before exporting to markdown, or to implement a custom serializer if you need more control over the output [source].

If you want to ensure consistent markdown export until this is addressed, use the in-memory re-validation step after conversion. If you need picture descriptions in markdown by default, customizing the serializer is the most robust long-term solution.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[vagenas]

Hi @lakatosd, this should now be fixed with docling-core v2.51.0, just upload your docling-core dependency (or install docling in a fresh environment).