-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Open
Labels
Description
Environment
- How did you deploy Kubeflow Pipelines (KFP)? Vertex AI Pipelines (+ MRE local)
- KFP version: 2.13.0
- KFP SDK version: kfp 2.13.0
Steps to reproduce
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = ["kfp==2.13.0"]
# ///
"""MRE: KFP pipelines with artifact inputs compile but can't be invoked directly."""
from kfp import compiler, dsl, local
from kfp.dsl import Dataset
@dsl.component
def process(data: dsl.Input[Dataset]) -> str:
return "done"
@dsl.pipeline
def my_pipeline(input_data: Dataset) -> str:
return process(data=input_data).output
if __name__ == "__main__":
# Compilation works
compiler.Compiler().compile(my_pipeline, "/tmp/pipeline.yaml")
print("Compilation: SUCCESS")
# Direct invocation fails
local.init(runner=local.SubprocessRunner(use_venv=False))
try:
my_pipeline(input_data=Dataset(uri="gs://bucket/data"))
print("Invocation: SUCCESS")
except ValueError as e:
print(f"Invocation: FAILED\n{e}")Output:
Compilation: SUCCESS
Invocation: FAILED
Argument input_data must be a parameter type, such as str, int, float, bool, dict, and list. Artifact inputs are not supported.
Expected result
Pipelines with artifact inputs should be invocable directly, like components. Currently:
- Compilation works
- Nesting in other pipelines works (PR feat(backend): implement subdag output resolution #11196)
- Direct invocation fails
This forces users to either duplicate pipeline logic or use dsl.importer with string URIs, losing type safety.
Materials and Reference
- [backend] <Artifacts are not accepted as pipeline Parameters> #9100 - Original issue (closed)
- feat(backend): implement subdag output resolution #11196 - Fixed nested pipeline artifact outputs (Oct 2024)
- Artifacts docs show
my_pipeline(dataset: Dataset)syntax that doesn't work at runtime
Impacted by this bug? Give it a 👍.