ibm_is_instance_group: waitForHealthyInstanceGroup treats 'unhealthy' as terminal, failing applies the API already accepted

[cchristous]

Description

waitForHealthyInstanceGroup in ibm/service/vpc/resource_ibm_is_instance_group.go declares:

healthStateConf := &resource.StateChangeConf{
    Pending: []string{SCALING},
    Target:  []string{HEALTHY},
    ...
}

An instance group reports unhealthy while a member is being added, replaced, or is still booting. Since unhealthy is in neither Pending nor Target, StateChangeConf classifies it as an unexpected state and returns as soon as it observes one. Terraform then fails the apply even though the API call succeeded and the group converges to healthy shortly afterwards.

This is not the timeout expiring, so raising timeouts does not help — the watcher gives up on first sight of the state rather than at a deadline. We have seen the same class of change abort after ~40s on one occasion and after ~5m10s on another.

Impact

waitForHealthyInstanceGroup is shared, so this affects:

• ibm_is_instance_group — create (:286), update (:405) and the pre-delete wait (:574)
• ibm_is_instance_group_manager
• ibm_is_instance_group_manager_policy
• ibm_is_instance_group_manager_action

Any change that adds or cycles members can trip it: instance_count, subnets, or an instance_template swap. The practical effect is a red pipeline on a change that actually succeeded, requiring a manual re-run.

Reproduction

1. Create an ibm_is_instance_group whose members take more than a few seconds to boot and become healthy.
2. Change instance_count (or subnets, or the instance_template).
3. terraform apply fails with:

Error: waitForHealthyInstanceGroup failed: unexpected state 'unhealthy',
wanted target 'healthy'. last error: %!s(<nil>)

4. The change has nonetheless been applied. The group reaches healthy on its own a short time later — in our case about two minutes after Terraform gave up — and re-running apply with an unchanged configuration succeeds.

Expected behaviour

unhealthy should be a pending state: keep polling until the group is healthy or the configured timeout expires. That is what the timeouts block is for, and it matches how transient states are handled elsewhere in the VPC resources. A group that genuinely cannot become healthy would then fail at the timeout instead of immediately — a slower failure, but a correct one, and tunable per configuration.

Versions

• Provider: 1.89.0
• Also present on master as of this writing.

I have a small patch for this and will open a PR shortly.