`azurerm_data_protection_backup_policy_postgresql`: support `target_copy` property and its related updates by sinbai · Pull Request #24476 · hashicorp/terraform-provider-azurerm

sinbai · 2024-01-12T09:13:01Z

When supporting targetDataStoreCopySettings property, it was found that TF is currently hard-coded for default azure retention rule and source data store partial value for the lifecycles(Additionally, there is the fact that for the default Azure retention rules/Azure retention rule, the lifecycle may have multiple items instead of the current hard-coded single item). The service team has confirmed that these properties are not immutable. I assume that these properties should be specified by the user.

Therefore, while submitting this PR to support targetDataStoreCopySettings, the original implementation (dependency) was changed and some properties were opened to users.

Test results:

Bug Fix
New Feature (ie adding a service, resource, or data source)
Enhancement
Breaking Change

katbyte

Thanks @sinbai - this looks like the new property won't take affect till 4.0?

is it possible to implement them now inline and support a graceful deprecation path for users instead of the hard breaking change in 4.0? as well allowing them to be used before?

sinbai · 2024-03-04T04:03:11Z

Thanks @sinbai - this looks like the new property won't take affect till 4.0?

is it possible to implement them now inline and support a graceful deprecation path for users instead of the hard breaking change in 4.0? as well allowing them to be used before?

Hi @katbyte thanks for your feedback and sounds like good suggestion. However, the newly added properties default_retention_rule and retention_rule.#.life_cycle should be required properties. If the original required attributes default_retention_duration and retention_rule.#.duration and the new two required are supported at the same time in the 3.0, these two sets of mutually exclusive attributes need to be modified to be optional. Then when the deprecated properties are deleted in the 4.0 , the new properties need to be changed to required. I assume this will be a breaking change (the new properties are changed from optional to required), so I assume this not appropriate, what do you think?

katbyte · 2024-03-05T03:40:07Z

Changing something from optional -> required during a major version is just fine! so lets make it optional & enable people using these properties now with the 4.0 flag flipping it to how things should behave in 4.0

sinbai · 2024-03-07T06:13:59Z

Changing something from optional -> required during a major version is just fine! so lets make it optional & enable people using these properties now with the 4.0 flag flipping it to how things should behave in 4.0

Hi @katbyte thanks for your time and feedback. I have updated the code. Could you please take another look? Test results are as follows:

katbyte · 2024-03-12T02:40:20Z

+
+---
+
+A `target_copy_setting` block supports the following:


I think setting is redundant?

Suggested change

A `target_copy_setting` block supports the following:

A `target_copy` block supports the following:

katbyte · 2024-03-12T02:40:53Z

+
+A `target_copy_setting` block supports the following:
+
+* `copy_option` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.


and this might be better as

Suggested change

* `copy_option` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.

* `option_json` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.

?

katbyte · 2024-03-12T02:41:35Z

+			"default_retention_duration",
+			"default_retention_rule.#",
+			"default_retention_rule.0.%",
+			"default_retention_rule.0.life_cycle.#",
+			"default_retention_rule.0.life_cycle.0.%",
+			"default_retention_rule.0.life_cycle.0.data_store_type",
+			"default_retention_rule.0.life_cycle.0.duration",


are these not returned from the API? how come we are ignoring them?

The API returns their values. However, TF supports both default_retention_duration (deprecated in v4.0) and default_retention_rule, which are two mutually exclusive properties in v3.0. These two properties corresponding to the same property of the API. Therefore, when importing resource in v3.0, I assume that we have no way of knowing whether the default_retention_duration or default_retention_rule is specified in the config, so I ignore them. Please see the following code in resourceDataProtectionBackupPolicyPostgreSQLRead func for details. Other than that, could you please let me know if there is a better solution?

in the past when i have deprecated a property and replaced it with another property i've made them both computed and then just set them both in read so they have a valid property, and then only sent the value that exists/has been set in config off to the api in update/create.

is this possible with these fields to do that?

Hi @katbyte thank you very much, I think this is a very good way. But there is a little problem. If the deprecated and replaced properties are set in read at the same time, then the following checks need to be deleted at the same time. Do you think it's okay to remove the check?

Hi @katbyte a kind reminder in case you miss the above message.

sinbai · 2024-03-12T05:56:17Z

Hi @katbyte I have updated the code to fix the comments above. The test results are as follows, could you please take another look? Thanks for your time.

github-actions · 2024-04-15T06:02:43Z

This PR is being labeled as "stale" because it has not been updated for 30 or more days.

If this PR is still valid, please remove the "stale" label. If this PR is blocked, please add it to the "Blocked" milestone.

If you need some help completing this PR, please leave a comment letting us know. Thank you!

github-actions · 2024-06-10T06:03:18Z

This PR is being labeled as "stale" because it has not been updated for 30 or more days.

If this PR is still valid, please remove the "stale" label. If this PR is blocked, please add it to the "Blocked" milestone.

If you need some help completing this PR, please leave a comment letting us know. Thank you!

stephybun · 2024-07-31T12:16:20Z

@sinbai having taken another look, this will require some significant rework before this can go in.

Before we go down that path though I have a specific question which is:

Has the request to add support for target_copy come from the service team or is this our own initiative to improve coverage here?

I ask because when creating a Backup Policy for PostgreSQL in the Portal, the option target copy setting isn't exposed at all and there is no way to set that information in the policy currently.

See the JSON excerpt below of the Backup Policy created via the portal:

{
    "properties": {
        "policyRules": [
            {
                "lifecycles": [
                    {
                        "deleteAfter": {
                            "objectType": "AbsoluteDeleteOption",
                            "duration": "P6M"
                        },
                        "targetDataStoreCopySettings": [],
                        "sourceDataStore": {
                            "dataStoreType": "VaultStore",
                            "objectType": "DataStoreInfoBase"
                        }
                    }
                ],
                "isDefault": false,
                "name": "Monthly",
                "objectType": "AzureRetentionRule"

sinbai · 2024-08-06T10:18:27Z

@sinbai having taken another look, this will require some significant rework before this can go in.

Before we go down that path though I have a specific question which is:

Has the request to add support for target_copy come from the service team or is this our own initiative to improve coverage here?

I ask because when creating a Backup Policy for PostgreSQL in the Portal, the option target copy setting isn't exposed at all and there is no way to set that information in the policy currently.

See the JSON excerpt below of the Backup Policy created via the portal:
{
    "properties": {
        "policyRules": [
            {
                "lifecycles": [
                    {
                        "deleteAfter": {
                            "objectType": "AbsoluteDeleteOption",
                            "duration": "P6M"
                        },
                        "targetDataStoreCopySettings": [],
                        "sourceDataStore": {
                            "dataStoreType": "VaultStore",
                            "objectType": "DataStoreInfoBase"
                        }
                    }
                ],
                "isDefault": false,
                "name": "Monthly",
                "objectType": "AzureRetentionRule"

Hi @stephybun sorry for the late reply. Yes, support for target_copy is a request from the service team. Also, I confirmed with the service team the necessity of this feature support and just received a positive response from them. Regarding the issue of not being visible via Portal, in fact, the feature is visible to some users/ vaults, see the picture below, and they will internally check the reason behind this.

sinbai · 2024-11-07T06:31:20Z

@sinbai as 4.0 has been released could we update the PR with 5.0 flags?

addtionally i think the schema can be simplified

Hi @katbyte I have fixed the PR with 5.0 flags. Could you please take another look?

katbyte · 2024-11-07T21:42:31Z

@sinbai as per discussed on tuesday here is the breaking change & deprecation guide that this PR should follow: https://github.com/hashicorp/terraform-provider-azurerm/commits/main/contributing/topics/guide-breaking-changes.md

sinbai · 2024-11-08T06:34:13Z

@sinbai as per discussed on tuesday here is the breaking change & deprecation guide that this PR should follow: https://github.com/hashicorp/terraform-provider-azurerm/commits/main/contributing/topics/guide-breaking-changes.md

Hi @katbyte , thanks for your time. I'm assuming I've followed the breaking changes and deprecation guidelines for now, please let me know if I've missed/misunderstood something. Thank you!

Test results in TC:

jackofallops

Hi @sinbai - Sorry for the delay on this.

Can you update the title of this PR to reflect the change, it doesn't appear to have anything to do with target_copy_setting, rather you're adding an entire block and deprecating a string property, to add support for target_copy.

Additionally, there are some important points to take a look at below if you could. Once these are addressed, I'll loop back to continue review.

Thanks

jackofallops · 2025-02-18T07:25:13Z

+		CustomizeDiff: pluginsdk.CustomizeDiffShim(func(ctx context.Context, diff *pluginsdk.ResourceDiff, v interface{}) error {
+			if !features.FivePointOhBeta() {
+				retentionRules := diff.Get("retention_rule")
+				defaultRetentionDuration := diff.Get("default_retention_duration")
+				defaultRetentionRule := diff.Get("default_retention_rule")
+
+				for i, rule := range retentionRules.([]interface{}) {
+					v := rule.(map[string]interface{})
+					if v["duration"].(string) == "" && len(v["life_cycle"].([]interface{})) == 0 || v["duration"].(string) != "" && len(v["life_cycle"].([]interface{})) > 0 {
+						return fmt.Errorf(`one of "retention_rule.%s.duration", "retention_rule.%s.life_cycle" must be specified`, strconv.Itoa(i), strconv.Itoa(i))
+					}
+
+					if defaultRetentionDuration != "" && v["duration"].(string) == "" {
+						return fmt.Errorf(`"default_retention_duration", "retention_rule.%s.duration" must be specified at the same time`, strconv.Itoa(i))
+					}
+
+					if len(defaultRetentionRule.([]interface{})) > 0 && len(v["life_cycle"].([]interface{})) == 0 {
+						return fmt.Errorf(`"default_retention_rule", "retention_rule.%s.life_cycle" must be specified at the same time`, strconv.Itoa(i))
+					}
+				}
+			}
+			return nil
+		}),


Can we move this into the feature flagged block below, it will make it easier and safer to remove after the 5.0 release.

jackofallops · 2025-02-18T07:33:57Z

+			ForceNew:      true,
+			Computed:      true,
+			ConflictsWith: []string{"default_retention_rule"},
+			Deprecated:    "`default_retention_duration` should be removed in favour of the `default_retention_rule.0.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",


Grammar, and formatting as per the deprecation guide.

Suggested change

Deprecated: "`default_retention_duration` should be removed in favour of the `default_retention_rule.0.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",

Deprecated: "`default_retention_duration` has been deprecated in favour of the `default_retention_rule.0.life_cycle.#.duration` and will be removed in version 5.0 of the AzureRM Provider.",

jackofallops · 2025-02-18T07:34:30Z

+			Optional:     true,
+			ForceNew:     true,
+			Computed:     true,
+			Deprecated:   "`retention_rule.#.duration` should be removed in favour of the `retention_rule.#.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",


Grammar, and formatting as per the deprecation guide.

Suggested change

Deprecated: "`retention_rule.#.duration` should be removed in favour of the `retention_rule.#.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",

Deprecated: "`retention_rule.#.duration` has been deprecated in favour of the `retention_rule.#.life_cycle.#.duration` and will be removed in version 5.0 of the AzureRM Provider.",

jackofallops · 2025-02-18T07:36:36Z

+		}),
 	}
+
+	if !features.FivePointOhBeta() {


This function has been removed, can you any instances update to use features.FivePointOh() please.

jackofallops · 2025-02-18T07:52:37Z

+	if v, ok := d.GetOk("default_retention_duration"); ok && !features.FivePointOhBeta() {
+		policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultAzureRetentionRule(v))
+		policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRuleArray(d.Get("retention_rule").([]interface{}))...)
+	} else {
+		policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultRetentionRule(d.Get("default_retention_rule").([]interface{})))
+		policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRules(d.Get("retention_rule").([]interface{}))...)
+	}


I think this can be changed to make it easier to remove when 5.0 ships:

Suggested change

if v, ok := d.GetOk("default_retention_duration"); ok && !features.FivePointOhBeta() {

policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultAzureRetentionRule(v))

policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRuleArray(d.Get("retention_rule").([]interface{}))...)

} else {

policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultRetentionRule(d.Get("default_retention_rule").([]interface{})))

policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRules(d.Get("retention_rule").([]interface{}))...)

}

if !features.FivePointOh() {

if v, ok := d.GetOk("default_retention_duration"); ok {

policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultAzureRetentionRule(v))

policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRuleArray(d.Get("retention_rule").([]interface{}))...)

} else {

policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultRetentionRule(d.Get("default_retention_rule").([]interface{})))

policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRules(d.Get("retention_rule").([]interface{}))...)

}

} else {

policyRules = append(policyRules, expandBackupPolicyPostgreSQLDefaultRetentionRule(d.Get("default_retention_rule").([]interface{})))

policyRules = append(policyRules, expandBackupPolicyPostgreSQLAzureRetentionRules(d.Get("retention_rule").([]interface{}))...)

}

jackofallops · 2025-02-18T07:59:42Z

+				if _, ok := d.GetOk("default_retention_duration"); ok && !features.FivePointOhBeta() {
+					if err := d.Set("default_retention_duration", flattenBackupPolicyPostgreSQLDefaultRetentionRuleDuration(&props.PolicyRules)); err != nil {
+						return fmt.Errorf("setting `default_retention_duration`: %+v", err)
+					}
+					if err := d.Set("retention_rule", flattenBackupPolicyPostgreSQLRetentionRuleArray(&props.PolicyRules)); err != nil {
+						return fmt.Errorf("setting `retention_rule`: %+v", err)
+					}
+				} else {
+					if err := d.Set("default_retention_rule", flattenBackupPolicyPostgreSQLDefaultRetentionRule(&props.PolicyRules)); err != nil {
+						return fmt.Errorf("setting `default_retention_rule`: %+v", err)
+					}
+					if err := d.Set("retention_rule", flattenBackupPolicyPostgreSQLRetentionRules(&props.PolicyRules)); err != nil {
+						return fmt.Errorf("setting `retention_rule`: %+v", err)
+					}
 				}


This will break importing the resource in almost all cases (which is why you had to ignore more or less everything in the test below). This will need to be reworked to allow those tests to pass correctly without ignoring the import errors.

jackofallops · 2025-02-18T08:10:02Z

 func TestAccDataProtectionBackupPolicyPostgreSQL_basic(t *testing.T) {
 	data := acceptance.BuildTestData(t, "azurerm_data_protection_backup_policy_postgresql", "test")
 	r := DataProtectionBackupPolicyPostgreSQLResource{}
+	if !features.FivePointOhBeta() {


These tests need to be split into 4.x and 5.0 versions, since they have differing functionality, and currently they will require significant effort to resolve/make right when 5.0 ships.

sinbai · 2025-02-19T09:31:37Z

Hi @sinbai - Sorry for the delay on this.

Can you update the title of this PR to reflect the change, it doesn't appear to have anything to do with target_copy_setting, rather you're adding an entire block and deprecating a string property, to add support for target_copy.

Additionally, there are some important points to take a look at below if you could. Once these are addressed, I'll loop back to continue review.

Thanks

Hi @jackofallops thanks for your feedback. I have updated the code, could you please take another look?

teowa · 2026-04-09T08:14:58Z

Close this as azurerm_data_protection_backup_policy_postgresql has been deprecated.

github-actions Bot added service/data-protection size/XL labels Jan 12, 2024

stephybun reviewed Jan 22, 2024

View reviewed changes

Comment thread internal/services/dataprotection/data_protection_backup_policy_postgresql_resource.go Outdated

Comment thread internal/services/dataprotection/data_protection_backup_policy_postgresql_resource.go Outdated

github-actions Bot added the waiting-response label Jan 22, 2024

github-actions Bot removed the waiting-response label Feb 17, 2024

katbyte requested changes Feb 27, 2024

View reviewed changes

katbyte added the waiting-response label Feb 27, 2024

github-actions Bot removed the waiting-response label Mar 4, 2024

katbyte added the waiting-response label Mar 5, 2024

sinbai force-pushed the dataprotection/support_new_properties branch from 8d946c6 to ed5ce33 Compare March 7, 2024 03:56

github-actions Bot added documentation and removed waiting-response labels Mar 7, 2024

katbyte requested changes Mar 12, 2024

View reviewed changes

github-actions Bot added the waiting-response label Mar 12, 2024

katbyte reviewed Mar 12, 2024

View reviewed changes

github-actions Bot removed the waiting-response label Mar 12, 2024

github-actions Bot added the waiting-response label Mar 12, 2024

github-actions Bot added the stale label Apr 15, 2024

github-actions Bot removed the stale label May 6, 2024

github-actions Bot added the stale label Jun 10, 2024

github-actions Bot removed the stale label Jul 31, 2024

github-actions Bot removed the waiting-response label Aug 6, 2024

jackofallops requested changes Feb 18, 2025

View reviewed changes

github-actions Bot assigned jackofallops Feb 18, 2025

github-actions Bot added the waiting-response label Feb 18, 2025

sinbai changed the title ~~azurerm_data_protection_backup_policy_postgresql: support target_copy_setting property~~ azurerm_data_protection_backup_policy_postgresql: support target_copy property and its related updates Feb 19, 2025

github-actions Bot added service/load-balancers service/storage service/dns service/hdinsight service/mssql Microsoft SQL Server service/event-hubs EventHubs service/traffic-manager service/service-bus service/search and removed waiting-response labels Feb 19, 2025

sinbai and others added 11 commits September 3, 2025 09:48

support target_copy_setting property

dc49361

update code

c3334ed

update code

187eaeb

update code

cc73eb3

fix comments

c82213a

fix comments

a3a3643

update doc

1f03595

fix comments

81216fd

fix comments

9f35272

fix comments

ff84bfd

update test

21a60b4

	A `target_copy_setting` block supports the following:
	A `target_copy` block supports the following:


		A `target_copy_setting` block supports the following:

		* `copy_option` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.

	* `copy_option` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.
	* `option_json` - (Required) Specifies when the backups are tiered across two or more selected data stores as a json encoded string. Changing this forces a new Backup Policy PostgreSQL to be created.

	Deprecated: "`default_retention_duration` should be removed in favour of the `default_retention_rule.0.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",
	Deprecated: "`default_retention_duration` has been deprecated in favour of the `default_retention_rule.0.life_cycle.#.duration` and will be removed in version 5.0 of the AzureRM Provider.",

	Deprecated: "`retention_rule.#.duration` should be removed in favour of the `retention_rule.#.life_cycle.#.duration` property in version 5.0 of the AzureRM Provider.",
	Deprecated: "`retention_rule.#.duration` has been deprecated in favour of the `retention_rule.#.life_cycle.#.duration` and will be removed in version 5.0 of the AzureRM Provider.",

Conversation

sinbai commented Jan 12, 2024 • edited by rcskosir Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

katbyte left a comment

Choose a reason for hiding this comment

Uh oh!

sinbai commented Mar 4, 2024

Uh oh!

katbyte commented Mar 5, 2024

Uh oh!

sinbai commented Mar 7, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sinbai commented Mar 12, 2024

Uh oh!

github-actions Bot commented Apr 15, 2024

Uh oh!

github-actions Bot commented Jun 10, 2024

Uh oh!

stephybun commented Jul 31, 2024

Uh oh!

sinbai commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sinbai commented Nov 7, 2024

Uh oh!

katbyte commented Nov 7, 2024

Uh oh!

sinbai commented Nov 8, 2024

Uh oh!

jackofallops left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sinbai commented Jan 12, 2024 •

edited by rcskosir

Loading

sinbai commented Aug 6, 2024 •

edited

Loading