prefect_deployment: pull_steps type too restrictive and cannot use custom importable path

[bdalpe]

I'm using yamldecode to import a prefect.yaml file and convert it into a prefect_deployment resource. Unfortunately, the pull_steps validation seems too restrictive and won't let me use the prefect.yaml syntax directly.

I can work around this issue by applying replace(type, "prefect.deployment.steps.", ""). This works for my use case, but the TF provider should have parity with prefect.yaml file syntax.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Terraform Version

OpenTofu v1.9.0
on darwin_arm64

• provider registry.opentofu.org/prefecthq/prefect v2.92.1

Affected Resource(s)

• prefect_deployment

Terraform Configuration Files

resource "prefect_flow" "hello_world" {
  name = "hello-world"
}

resource "prefect_deployment" "demos" {
  name = "demo"
  entrypoint = "example.py:hello_world"
  flow_id = prefect_flow.hello_world

  pull_steps = [
    {
      "type": "prefect.deployments.steps.git_clone",
      "repository": "https://github.com/PrefectHQ/examples.git"
    }
  ]
}

Expected Behavior

Should accept any string.

See: https://docs.prefect.io/v3/how-to-guides/deployments/prefect-yaml#deployment-actions

Actual Behavior

│ Error: Invalid Attribute Value Match
│ 
│   with prefect_deployment.demos["crypto_prices/crypto-prices-daily"],
│   on main.tf line 76, in resource "prefect_deployment" "demos":
│   76: resource "prefect_deployment" "demos" {
│ 
│ Attribute pull_steps[0].type value must be one of: ["set_working_directory" "git_clone" "pull_from_azure_blob_storage" "pull_from_gcs" "pull_from_s3"], got: "prefect.deployments.steps.git_clone"

Steps to Reproduce

1. terraform apply

[mitchnielsen]

I see where you're coming from on this one. I wanted to provide some guardrails here because it allows us to enumerate the available options on an already pretty complex object, and do some validation on the provided value in case someone makes a typo etc.

If these options changed frequently, or if they had many potential values, I'd be more inclined to leave this as a plain text field. But with a seemingly limited number of options, I do like that users can see what options they have and easily pick one. From what I saw so far, the documentation doesn't clearly lay out the list of available options but I may have just missed it.

What do you think?

[bdalpe]

@mitchnielsen I definitely see the value in having strongly typed objects with validation.

But the step definition allows for custom importable steps and I think we should allow someone to be able to define this in the deployment resource (but making known that we won't validate the schema for any custom usage):

Steps are assumed to be in the format {"importable.func.name": {"kwarg1": "value1", ...}}.

https://github.com/PrefectHQ/prefect/blob/6645cd63c2e36f1a342d0f39de01cf4b3dd1e045/src/prefect/deployments/steps/core.py#L114

I wonder if there is a way to have the best of both worlds here...

[bdalpe]

I'd also like the resource pull_step types to support the fully importable names:

terraform-provider-prefect/internal/provider/resources/deployment.go

Lines 402 to 406 in 00a4ab0

	"set_working_directory",
	"git_clone",
	"pull_from_azure_blob_storage",
	"pull_from_gcs",
	"pull_from_s3",

We can keep the existing ones for backwards compatibility, but add the full paths to this list.

[mitchnielsen]

We can keep the existing ones for backwards compatibility, but add the full paths to this list.

To make sure we're on the same page: that makes sense to me, but that still wouldn't enable the custom pull steps you mentioned, right?

If we want to support that, we could make it a separate configuration. Something like pull_steps + custom_pull_steps. We'd then keep the type checking for the pull_steps, and then any custom_pull_steps provided would be appended to any API requests as-is?

[mitchnielsen]

cc @bdalpe

[bdalpe]

Partially resolved by #655 -- added missing existing steps