XCom dictionary key order not preserved

[vshih]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

When passing a dictionary through XCom (at least as a return-value), key order is not preserved. The keys appear to be sorted alphabetically when inspected from a consuming task.

What you think should happen instead?

Since Python 3.7, dictionaries preserve the insertion order of keys. JSON dumping and loading likewise preserve this also. Since Airflow is well past that version, I would expect it to simply inherit this behavior.

How to reproduce

Minimal DAG:

from airflow.decorators import dag, task


@dag(render_template_as_native_obj=True)
def d1():
    @task
    def create() -> dict:
        return {
            'C': 1,
            'B': 2,
            'A': 3,
        }

    @task
    def render(d: dict):
        print(d)

    render(create())


d1()

The output:

INFO - {'A': 3, 'B': 2, 'C': 1} source=task.stdout

Operating System

MacOS Tahoe 26.0.1 (25A362)

Versions of Apache Airflow Providers

I doubt any of these are relevant, but just in case:

apache-airflow==3.1.0
apache-airflow-core==3.1.0
apache-airflow-providers-amazon==9.14.0
apache-airflow-providers-celery==3.12.3
apache-airflow-providers-cncf-kubernetes==10.8.1
apache-airflow-providers-common-compat==1.7.4
apache-airflow-providers-common-io==1.6.3
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-docker==4.4.3
apache-airflow-providers-elasticsearch==6.3.3
apache-airflow-providers-fab==2.4.3
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.8
apache-airflow-providers-google==18.0.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.4
apache-airflow-providers-microsoft-azure==12.7.1
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.1
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-redis==4.3.1
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.4.0
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-smtp==2.2.1
apache-airflow-providers-snowflake==6.5.4
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-standard==1.8.0
apache-airflow-task-sdk==1.1.0
Jinja2==3.1.6

Deployment

Docker-Compose

Deployment details

N/A (fresh docker install)

Anything else?

Happens 100% of the time.

Some possible clues:

The "create" task has this in its log:

Done. Returned value was: {'C': 1, 'B': 2, 'A': 3} source=airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator loc=python.py:218

The order seems to be preserved up to this point. However, in its XCom pane, the return_value shows the dictionary with its keys sorted:

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

To me it is not a bug, we never promised to preserve the order of the dict keys. As you mentioned, since 3.7, Python started retains insertion order in dictionary, and json package inherited this behavior automatically because it uses native dictionary. However, JSON object is still unordered, so JavaScript and other languages can load/dump the object in any order, which I think the case in the UI (Typescript)

[hussein-awala]

I did some debugging and here is what I found. The XCom value lose the key orders in the DB:

select * from xcom where task_id='create' and key='return_value';

dag_run_id | task_id | map_index |     key      | dag_id |                  run_id                  |          value           |           timestamp           
------------+---------+-----------+--------------+--------+------------------------------------------+--------------------------+-------------------------------
          8 | create  |        -1 | return_value | d1     | manual__2025-10-11T09:43:25.379548+00:00 | {"A": 3, "B": 2, "C": 1} | 2025-10-11 09:43:26.917442+00
(1 row)

Why?
It's because we store the value as a JSON for all the DBs except for PostgreSQL where we use JSONB:

value = Column(JSON().with_variant(postgresql.JSONB, "postgresql"))

• JSON keeps insertion order of keys (the exact string representation) — slower to query, faster to insert.
• JSONB reorders keys arbitrarily and removes duplicates during storage — faster to query, slower to insert.

[vshih]

Interesting, thanks for the explanation.

I agree that it's not a bug, however to me this is unexpected as it is incongruent with promises in the underlying technology.

I wonder if this could be an option to switch somewhere. In fact it seems that whatever performance gain would be minimal - I guess in cases where the data is read many times? I would vote that the default behavior be the more compatible/expected one, and that users could opt-in to the more performant behavior.

[hussein-awala]

I think the general community answer (including mine) would be: XCom is extensible, if you need a feature that may not be useful to others, you can create your own backend using a database or any other storage system.

[jroachgolf84]

Just weighing in here; I don't see a huge upside for maintaining the order of these keys. In fact, it might actually make sense for it to be "alphabetized" to provide UX consistency. I'll leave it up to other folks, however.

[potiuk]

Since it is "remote" and potentially XCom is usable in other languages (soon) - we actually even should not preserve the sequence "as in Python" - because it might be difficult or even impossible in other languages where dictionary might be optimized and might not preserve sequence (typical HashMap behaviour is random ordering).

I think we should not make any promise about ordering here.