bigquerydatatransfer: add schedule_options_v2 to google_bigquery_data_transfer_config

[alien2003]

Fixes hashicorp/terraform-provider-google#24274

google_bigquery_data_transfer_config had no way to configure event-driven (Pub/Sub) schedules, and more generally no mapping for the API's scheduleOptionsV2.

This adds a schedule_options_v2 block that mirrors the API oneof:

• time_based_schedule (schedule, start_time, end_time)
• manual_schedule (empty block; runs only via StartManualTransferRuns)
• event_driven_schedule (pubsub_subscription), the option asked for in the issue, supported for the Cloud Storage data source

Implementation notes:

• The three sub-blocks form an exactly_one_of group, matching the API.
• Per the API, scheduleOptionsV2 replaces the legacy schedule/schedule_options, so they aren't meant to be combined. I left that to API-side validation rather than a schema ConflictsWith, since ConflictsWith across nested blocks is unreliable in the SDK.
• This resource builds its update mask by hand in pre_update, so I added a scheduleOptionsV2 entry there; without it, updates to the block would be dropped.
• An earlier attempt (Fix : google_bigquery_data_transfer_config should support EventDrivenSchedule  #15094) placed event_driven_schedule under schedule_options. Per the v1 discovery document the field belongs to scheduleOptionsV2, which is what this PR uses.

Testing:

• make provider VERSION=ga PRODUCT=bigquerydatatransfer, then go build, go vet, and go test for the package and Provider().InternalValidate() all pass.
• Added a handwritten acceptance test (schedule_options_v2 with time_based_schedule) modeled on the existing scheduled-query test. I don't have a billing-enabled project so I have not run the acc suite; please run it via CI. The event-driven path needs a configured Cloud Storage source plus Pub/Sub IAM, so its example is exclude_test, consistent with the other examples on this resource.
• One thing worth confirming on the acc run: the read-back interaction between schedule_options_v2 and the legacy schedule/schedule_options fields (whether the server echoes a value into the legacy fields when V2 is used).

Disclosure: drafted with help from Claude Code; the diff and the API mapping (against the v1 discovery document) were reviewed manually.

bigquerydatatransfer: added `schedule_options_v2` field to `google_bigquery_data_transfer_config`

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[github-actions]

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@melinath, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[melinath]

Thanks for the PR! I haven't thoroughly reviewed this yet but the test failures look like this:

=== RUN   TestAccBigqueryDataTransferConfig/service_account
    resource_bigquery_data_transfer_config_test.go:485: Step 1/2 error: After applying this test step, the non-refresh plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # google_bigquery_data_transfer_config.query_config will be updated in-place
          ~ resource "google_bigquery_data_transfer_config" "query_config" {
                id                        = "projects/1067888929963/locations/asia-northeast1/transferConfigs/6a3dd3a7-0000-2f99-bebf-34c7e91f51eb"
                name                      = "projects/1067888929963/locations/asia-northeast1/transferConfigs/6a3dd3a7-0000-2f99-bebf-34c7e91f51eb"
                # (12 unchanged attributes hidden)
        
              - schedule_options_v2 {
                  - time_based_schedule {
                      - schedule   = "every day 00:00" -> null
                        # (2 unchanged attributes hidden)
                    }
                }
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.

I'd interpret this as: the server probably returns a value if nothing is set in the config. See https://googlecloudplatform.github.io/magic-modules/develop/diffs/ for guidance on resolving this (most likely via default_from_api.)

This is also missing a test for the new pubsub_subscription field.

[github-actions]

@alien2003, this PR is waiting for action from you. If no action is taken, this PR will be closed in 28 days.

Please address any comments or change requests, or re-request review from a core reviewer if no action is required.

This notification can be disabled with the disable-automatic-closure label.

[alien2003]

@melinath

1. Non-empty plan on service_account and update_service_account - fixed by marking scheduleOptionsV2 as default_from_api (Optional + Computed)
2. Missing test for pubsub_subscription - added an acceptance test (schedule_options_v2_event_driven)

Regenerated the GA provider and confirmed schedule_options_v2 now generates as Optional + Computed; go vet, go build, Provider().InternalValidate() (TestProvider). I currently don't have a billing-enabled project I can test this on, so I haven't run the acc suite, TestAccBigqueryDataTransferConfig/schedule_options_v2 and .../schedule_options_v2_event_driven still need to be tested

[melinath]

We don't want to provision IAM resources in these tests. I see that this isn't unique to this test and you probably copied it from one of the other tests, but we shouldn't be doing it & we should actually remove it from all the tests. The problem is that managing IAM in this test could cause other tests relying on that IAM to fail; there can also be failures caused by propagation delays.

See https://googlecloudplatform.github.io/magic-modules/test/test/#iam-resources (the "Handwritten" tab for information about how to bootstrap IAM permissions; this should probably happen once in the parent test & be taken out of the child test configs.

[alien2003]

removed the shared Data Transfer service-agent bindings, bootstrap them once in the parent TestAccBigqueryDataTransferConfig instead

[melinath]

Current test failure is:

Error: Error creating Config: googleapi: Error 400: Cannot find the destination table provided in request: my_table in dataset my_datasetksxzuvyo3b

This is happening when trying to create the google_bigquery_data_transfer_config.

It looks like this test config doesn't have a table defined. You'll also need to have an explicit dependency on it to ensure the creation order is right.

Suggested change

	destination_table_name_template = "my_table"
	destination_table_name_template = google_bigquery_table.my_table.table_id

[alien2003]

added google_bigquery_table.my_table, pointed destination_table_name_template -> google_bigquery_table.my_table.table_id

[melinath]

same comment here re: IAM bootstrapping.

[alien2003]

bootrapped once in the parent test, IAM removed from this config

[melinath]

O+C doesn't just save the value; it will also send it on subsequent requests. If both old & new values are sent in an API request, what happens? If that causes problems, this may need a different solution to suppress the diff without setting a value for the fields.

I see that testAccBigqueryDataTransferConfig_scheduledQuery_update does an update test with a scheduled_options block; in theory that should catch if there were any problems but I may want to manually verify just to be safe.

[alien2003]

I switched to ignore_read per the diffs guide. The field wasn't populated on import, so I added it to ImportStateVerifyIgnore in the new tests

[melinath]

Please add update steps to the new tests to make sure all updatable fields get exercised: https://googlecloudplatform.github.io/magic-modules/test/test/#add-an-acceptance-test (the Handwritten tab.) TLDR the main thing is to have a second step that has a new config updating the fields, using ConfigPlanChecks to make sure that the resource isn't being replaced.

[alien2003]

added a second step to both tests that updates an in-place field, the schedule for time-based, the Pub/Sub subscription for event-driven to catch any accidental recreation

[alien2003]

Validated locally: regenerated the GA provider and confirmed schedule_options_v2 is now Optional only (no Computed) and Read no longer sets it; gofmt, go vet, go build ./..., the bigquerydatatransfer unit tests, and TestProvider (InternalValidate) all pass. I still don't have a billing-enabled project for the acc suite, relying on the VCR recording for schedule_options_v2_event_driven; the time-based test uses time.Now so it stays VCR-skipped and will run in nightly. (All in 59b40e5.)

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit 59b40e5:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository	Diff Link	Changes
google provider	View Diff	4 files changed, 550 insertions(+), 46 deletions(-)
google-beta provider	View Diff	4 files changed, 550 insertions(+), 46 deletions(-)
terraform-google-conversion	View Diff	1 file changed, 134 insertions(+)

Test report

Analytics

Total Tests	Passed	Skipped	Affected
1	0	0	1

Affected Service Packages

• bigquerydatatransfer

Learn how VCR tests work

---

Step 1: Replaying Mode

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit.

Click here to see the affected tests

• TestAccBigqueryDataTransferConfig

View the replaying VCR build log

---

Step 2: Recording Mode

Recording Mode	Replaying Rerun	Test Name
✅ Log	✅	TestAccBigqueryDataTransferConfig/booleanParam
✅ Log	✅	TestAccBigqueryDataTransferConfig/schedule_options_v2_event_driven
✅ Log	✅	TestAccBigqueryDataTransferConfig/service_account
✅ Log	✅	TestAccBigqueryDataTransferConfig/update_params
✅ Log	✅	TestAccBigqueryDataTransferConfig/update_service_account

🟢 All tests passed!

View the recording VCR build log or the debug logs folder for detailed results.

@alien2003, @melinath, @sachinpro VCR tests complete for 59b40e5!

[melinath]

Manual test run to verify scheduleOptionsV2 behavior (you won't have access to this): https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_BIGQUERYDATATRANSFER/688540

[melinath]

looking through the test runs for this resources, I see the following behaviors:

• Setting schedule and/or scheduleOptions in the request results in the API returning scheduleOptions and scheduleOptionsV2 (where scheduleOptionsV2 is the combination of both fields)
  • if only schedule is sent, scheduleOptions will be returned as an empty object.
• Setting scheduleOptionsV2.eventDrivenSchedule doesn't set schedule or scheduleOptions
• BUT setting scheduleOptionsV2.timeBasedSchedule sets both schedule and scheduleOptions

The main point is that scheduleOptionsV2 is returned by the API and therefore shouldn't be marked ignore_read - doing so would be considered a bug (because users will encounter it as such in the future).

To solve the permadiffs, instead use diff suppression that checks what's set in the resource config & suppresses based on that, similar to #17709. For example, if schedule is set in the config, and scheduleOptionsV2.timeBasedSchedule.schedule isn't set in the config, suppress diffs on scheduleOptionsV2.timeBasedSchedule.schedule, and so on.

You'll need to add a test for disable_auto_scheduling=true (which I assume is going to be equivalent to scheduleOptionsV2.manualSchedule); doesn't look like we have one yet.

Something that would help with tests (if it would work): remove the SkipIfVcr calls from all the tests for this resource and instead ensure that the times are relatively static but always in the future - something like: time.Date(time.Now().Year(), 12, 31, 0, 0, 0, 0, time.Now().Location()).AddDate(0, 0, 10).Format(time.RFC3339). That helps avoid needing manual tests like the one above.

[melinath]

(Alternatively, you could remove support for scheduleOptionsV2.timeBasedSchedule and scheduleOptionsV2.manualSchedule. It would still be good to add a disable_auto_scheduling=true test.)

[alien2003]

@melinath, thank you for the review!

• you're right that scheduleOptionsV2 is readable, so ignore_read was wrong. Dropped time_based_schedule/manual_schedule and kept event_driven_schedule
• For the echo from legacy fields, added a decoder (like Fixed incorrectly-ignored read for google_compute_reservation.share_settings #17709) that drops scheduleOptionsV2 when it has no eventDrivenSchedule. Schema's Optional only now
• Added a disable_auto_scheduling=true test
• Switched to the static-future time form you suggested - basic/update/no_destination/CMEK now run under VCR

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit e51dc1c:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository	Diff Link	Changes
google provider	View Diff	4 files changed, 420 insertions(+), 65 deletions(-)
google-beta provider	View Diff	4 files changed, 420 insertions(+), 65 deletions(-)
terraform-google-conversion	View Diff	1 file changed, 54 insertions(+)

Test report

Analytics

Total Tests	Passed	Skipped	Affected
1	0	0	1

Affected Service Packages

• bigquerydatatransfer

Learn how VCR tests work

---

Step 1: Replaying Mode

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit.

Click here to see the affected tests

• TestAccBigqueryDataTransferConfig

View the replaying VCR build log

---

Step 2: Recording Mode

Recording Mode	Replaying Rerun	Test Name
✅ Log	✅	TestAccBigqueryDataTransferConfig/basic
✅ Log	✅	TestAccBigqueryDataTransferConfig/booleanParam
✅ Log	✅	TestAccBigqueryDataTransferConfig/disable_auto_scheduling
✅ Log	✅	TestAccBigqueryDataTransferConfig/no_destintation
✅ Log	✅	TestAccBigqueryDataTransferConfig/schedule_options_v2_event_driven
✅ Log	✅	TestAccBigqueryDataTransferConfig/service_account
✅ Log	✅	TestAccBigqueryDataTransferConfig/update
✅ Log	✅	TestAccBigqueryDataTransferConfig/update_params
✅ Log	✅	TestAccBigqueryDataTransferConfig/update_service_account

🟢 All tests passed!

View the recording VCR build log or the debug logs folder for detailed results.

@alien2003, @melinath, @sachinpro VCR tests complete for e51dc1c!