Skip to content

GCSPickleIOManager doesn't respect allow_missing_partitions with MultiToSingleDimensionPartitionMapping #32488

@mirthbottle

Description

@mirthbottle

What's the issue?

Description

When using MultiToSingleDimensionPartitionMapping with GCSPickleIOManager,
the allow_missing_partitions=True metadata is not respected when the upstream
multi-partitioned asset uses StaticPartitionsDefinition or DynamicPartitionsDefinition. This causes 404 errors
when some partitions don't exist.

Current Behavior

  • 404 NotFound error when trying to load missing partitions
  • allow_missing_partitions=True is ignored

What did you expect to happen?

Expected Behavior

  • Missing partitions should be skipped when allow_missing_partitions=True
  • Should return only the existing partitions in the Dict

How to reproduce?

Reproduction

import dagster as dg

@dg.asset(
    partitions_def=dg.MultiPartitionsDefinition({
        "date": dg.DailyPartitionsDefinition(start_date="2025-01-01"),
        "location": dg.StaticPartitionsDefinition(["A", "B"])
    }),
    io_manager_key="gcs_pickle_io"
)
def upstream_asset(context):
    ...

@dg.asset(
    ins={
        "upstream_asset": dg.AssetIn(
            partition_mapping=dg.MultiToSingleDimensionPartitionMapping(
                partition_dimension_name="date"
            ),
            metadata={"allow_missing_partitions": True}  # Not respected
        )
    }
)
def downstream_asset(context, upstream_asset):
    ...

Dagster version

1.11.13

Deployment type

Local

Deployment details

I am running dg dev locally

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions