Postgres Clickpipe must be paused to edit

[eudaimos]

Description

I get an error and failed terraform apply when attempting to add more tables to an existing Postgres Clickpipe.

I searched the issues first and could not find any relevant open or closed issues related to a Clickpipe needing to be paused ("pause" and "paused" search terms produce 0 results).

Cloud provider

Not applicable

Service tier

Not applicable

Affected resource(s)

clickhouse_clickpipe

Provider version

3.11.1-alpha1

Terraform version

1.11.1

Terraform configuration

It's pretty self-explanatory. I'm using Terragrunt v0.75.0 so I'm not sure how to produce the minimal TF config

Expected behavior

The TF Provider automates pausing the Clickpipe itself before performing the update.

Actual behavior

I get the following error from my CI/CD (Github Actions run):

  ╷
  │ Error: Error Updating ClickPipe
  │ 
  │   with clickhouse_clickpipe.postgres_clickpipes["pg_public_cdc"],
  │   on main.tf line 229, in resource "clickhouse_clickpipe" "postgres_clickpipes":
  │  229: resource "clickhouse_clickpipe" "postgres_clickpipes" {
  │ 
  │ Could not update ClickPipe, unexpected error: status: 400, body:
  │ {"requestId":"d3c7e480-84a0-4cd1-a877-e3397e4715af","error":"BAD_REQUEST:
  │ Postgres ClickPipe must be paused to edit","status":400}
  ╵

Additional context

In order to get it to pass and do what is expected, I have to use my browser and navigate to the Web Console > Service > Data Sources > Clickpipe > Settings and manually pause the clickpipe.

[tpanetti]

Hi @eudaimos, apologies that this is a little obtuse for the moment. ClickPipes will soon be in the GA provider release and have proper docs to explain all of this. For the moment, to pause your ClickPipe simply set stopped to true in your plan. After making the changes you need, to start the pipe again, set stopped to false

[eudaimos]

thanks @tpanetti - as a user I would not release that as GA since that requires 2 passes through my CI/CD to get the tables in. Nothing else in the TF provider work like that afaik. For most, the time consuming and laborious part of CI/CD is the CI part and I'll have to execute all of my CI steps/tests and go through PR checks then the same checks post-merge 2x just to add some tables (or remove tables) from my clickpipe

[morsapaes]

That's fair, @eudaimos, but not all updates require a pipe to be paused. In general, performing changes in state transparently isn't aligned with how Terraform works, as a declarative tool. The two-step pattern is common in other providers for resources with state dependencies; handling this as one step can lead to conflicts on concurrent access.

[eudaimos]

@morsapaes I respectfully disagree. I've worked with Terraform for a long time and greatly appreciate its declarative structure, one which led to the success of kubernetes which adopted the same structure. I've also worked on building both Terraform Providers and Kubernetes Operators. The introduction of the state of a clickpipe is recent and new to this provider and not a welcome one at that given that people have attempted to ignore that attribute to avoid seeing the unknown potential state change spit out by running plan on terraform declaration that has not changed.
I am personally not familiar with any TF Providers I've used in the past that require this 2-step dance to stop a process for it to receive changes declaratively and happy to review if you have some examples that I'm not presently aware of.
That being said, I consider this specific to developer ergonomics and DevX that I should not have to run a 2-pass operation to perform something as simple as adding a new table to my clickpipe. The purpose of the declarative language as I understand it is so that I as a developer declare the state of the system I want and delegate the inner logic of how to get there to the provider. Stopping and starting a clickpipe to accept a simple change of adding a new table is an implementation detail that should be hidden from me.

[morsapaes]

Stopping and starting a clickpipe to accept a simple change of adding a new table is an implementation detail that should be hidden from me.

Re-reading my own "not all updates require a pipe to be paused": I agree that this is a usability problem and the focus should maybe be on fixing that flow rather than discussing how to make it ergonomic in Terraform. Let me discuss this with the team again from a different angle.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove no-issue-activity label or comment or this will be closed in 5 days.

[dan-jacobson-artemis]

@morsapaes Any plans here? This pause/resume two-step is currently blocking us from refactoring to use the provider.