-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Open
Labels
type: bugSomething isn't workingSomething isn't working
Description
What's the issue?
Description
When using MultiToSingleDimensionPartitionMapping
with GCSPickleIOManager
,
the allow_missing_partitions=True
metadata is not respected when the upstream
multi-partitioned asset uses StaticPartitionsDefinition
or DynamicPartitionsDefinition
. This causes 404 errors
when some partitions don't exist.
Current Behavior
- 404 NotFound error when trying to load missing partitions
allow_missing_partitions=True
is ignored
What did you expect to happen?
Expected Behavior
- Missing partitions should be skipped when
allow_missing_partitions=True
- Should return only the existing partitions in the Dict
How to reproduce?
Reproduction
import dagster as dg
@dg.asset(
partitions_def=dg.MultiPartitionsDefinition({
"date": dg.DailyPartitionsDefinition(start_date="2025-01-01"),
"location": dg.StaticPartitionsDefinition(["A", "B"])
}),
io_manager_key="gcs_pickle_io"
)
def upstream_asset(context):
...
@dg.asset(
ins={
"upstream_asset": dg.AssetIn(
partition_mapping=dg.MultiToSingleDimensionPartitionMapping(
partition_dimension_name="date"
),
metadata={"allow_missing_partitions": True} # Not respected
)
}
)
def downstream_asset(context, upstream_asset):
...
Dagster version
1.11.13
Deployment type
Local
Deployment details
I am running dg dev
locally
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
Metadata
Metadata
Assignees
Labels
type: bugSomething isn't workingSomething isn't working