Skip to content

K8SPSMDB-1772: Wait for MCS service import hostname to resolve before proceeding#2482

Open
ravisingal wants to merge 1 commit into
percona:mainfrom
ravisingal:wait-for-mcs-dns-resolution
Open

K8SPSMDB-1772: Wait for MCS service import hostname to resolve before proceeding#2482
ravisingal wants to merge 1 commit into
percona:mainfrom
ravisingal:wait-for-mcs-dns-resolution

Conversation

@ravisingal

Copy link
Copy Markdown
Contributor

Summary

  • DNS propagation for imported multi-cluster (MCS) services can lag behind ServiceImport registration. The reconciler now verifies the imported service's hostname actually resolves before treating the replset as ready to proceed, instead of assuming DNS is immediately available once the ServiceImport object exists.
  • Adds psmdb.GetMCSHost as a shared helper for building the MCS FQDN (without port), and refactors GetMCSAddr to reuse it instead of duplicating the format string.

Test plan

  • go build ./...
  • gofmt -l / go vet clean on touched files
  • New unit tests: TestHostResolvable, TestReconcilerResolverDefault (controller), TestGetMCSHost (psmdb)
  • Existing unit tests in both touched packages pass

DNS propagation for imported multi-cluster services can lag behind
ServiceImport registration, so the reconciler now checks that the
hostname actually resolves before treating the import as ready.

Adds psmdb.GetMCSHost as a shared helper for building the MCS FQDN,
reused by GetMCSAddr.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@JNKPercona

Copy link
Copy Markdown
Collaborator
Test Name Result Time
arbiter passed 00:11:55
balancer failure 00:06:44
cert-management-policy failure 00:01:41
clustersync passed 00:14:39
cross-site-sharded passed 00:20:34
custom-replset-name passed 00:10:20
custom-tls passed 00:17:13
custom-users-roles passed 00:12:26
custom-users-roles-sharded passed 00:15:48
data-at-rest-encryption passed 00:14:46
data-sharded passed 00:24:28
demand-backup passed 00:23:56
demand-backup-eks-credentials-irsa passed 00:00:11
demand-backup-fs passed 00:26:58
demand-backup-if-unhealthy failure 00:01:42
demand-backup-incremental-aws failure 00:01:55
demand-backup-incremental-azure passed 00:11:08
demand-backup-incremental-gcp-native passed 00:11:27
demand-backup-incremental-gcp-s3 passed 00:11:43
demand-backup-incremental-oss-s3 passed 00:15:23
demand-backup-incremental-oss-native passed 00:16:12
demand-backup-incremental-minio passed 00:26:13
demand-backup-incremental-sharded-aws failure 00:22:21
demand-backup-incremental-sharded-azure passed 00:17:56
demand-backup-incremental-sharded-gcp-native passed 00:18:06
demand-backup-incremental-sharded-gcp-s3 passed 00:20:38
demand-backup-incremental-sharded-oss-s3 passed 00:28:35
demand-backup-incremental-sharded-oss-native passed 00:26:07
demand-backup-incremental-sharded-minio failure 00:08:17
demand-backup-logical-minio-native-tls passed 00:10:00
demand-backup-physical-parallel passed 00:10:36
demand-backup-physical-aws passed 00:12:24
demand-backup-physical-azure passed 00:14:51
demand-backup-physical-gcp-s3 passed 00:12:42
demand-backup-gcs-workload-identity passed 00:20:02
demand-backup-physical-gcp-native passed 00:12:27
demand-backup-physical-oss-s3 failure 00:16:49
demand-backup-physical-oss-native passed 00:15:26
demand-backup-physical-oci passed 00:13:34
demand-backup-physical-minio passed 00:22:55
demand-backup-physical-minio-native passed 00:27:23
demand-backup-physical-minio-native-tls passed 00:21:54
demand-backup-physical-sharded-parallel passed 00:11:43
demand-backup-physical-sharded-aws passed 00:19:45
demand-backup-physical-sharded-azure passed 00:21:10
demand-backup-physical-sharded-gcp-native passed 00:19:21
demand-backup-physical-sharded-oss-s3 passed 00:29:44
demand-backup-physical-sharded-oss-native passed 00:27:32
demand-backup-physical-sharded-minio passed 00:22:26
demand-backup-physical-sharded-minio-native passed 00:18:21
demand-backup-sharded passed 00:34:41
demand-backup-snapshot passed 00:46:33
demand-backup-snapshot-vault passed 00:18:47
disabled-auth passed 00:17:24
expose-sharded passed 00:34:44
external-dns passed 00:06:43
finalizer passed 00:10:33
ignore-labels-annotations passed 00:08:14
init-deploy passed 00:16:23
ldap passed 00:09:13
ldap-tls passed 00:13:13
limits passed 00:06:14
liveness passed 00:10:29
mongod-major-upgrade passed 00:13:15
mongod-major-upgrade-sharded passed 00:21:00
monitoring-2-0 passed 00:23:32
monitoring-pmm3 passed 00:48:40
multi-cluster-service passed 00:13:31
multi-storage passed 00:23:02
non-voting-and-hidden passed 00:22:03
one-pod passed 00:08:14
operator-self-healing-chaos passed 00:36:19
pitr passed 00:39:32
pitr-physical passed 00:59:45
pitr-sharded passed 00:20:14
pitr-to-new-cluster passed 00:28:41
pitr-physical-backup-source passed 00:52:33
preinit-updates passed 00:05:25
pvc-auto-resize passed 00:14:24
pvc-resize passed 00:16:50
recover-no-primary passed 00:29:03
replset-overrides passed 00:18:51
replset-remapping passed 00:19:01
replset-remapping-sharded passed 00:18:23
rs-shard-migration passed 00:13:59
scaling passed 00:16:57
scheduled-backup passed 00:22:00
security-context passed 00:07:03
self-healing-chaos passed 00:15:43
service-per-pod passed 00:23:34
serviceless-external-nodes passed 00:10:17
smart-update passed 00:08:27
split-horizon passed 00:14:32
split-horizon-manual-tls passed 00:13:13
stable-resource-version passed 00:04:43
storage passed 00:08:22
tls-clusterissuer-cert-manager passed 00:35:35
tls-issue-cert-manager passed 00:37:24
unsafe-psa passed 00:07:51
upgrade passed 00:14:09
upgrade-consistency passed 00:07:47
upgrade-consistency-sharded-tls failure 00:04:20
upgrade-sharded passed 00:21:14
upgrade-partial-backup passed 00:16:43
users failure 00:00:56
users-vault passed 00:13:37
vector-search passed 00:00:10
vector-search-sharded passed 00:00:08
version-service passed 00:25:14
Summary Value
Tests Run 109/109
Job Duration 04:17:16
Total Test Time 32:10:50

commit: 8b0df9c
image: perconalab/percona-server-mongodb-operator:PR-2482-8b0df9c16

@egegunes egegunes changed the title Wait for MCS service import hostname to resolve before proceeding K8SPSMDB-1772: Wait for MCS service import hostname to resolve before proceeding Jul 23, 2026
@egegunes egegunes added this to the v1.24.0 milestone Jul 23, 2026

// hostResolvable reports whether host resolves to at least one IP address.
// A resolver error is returned to the caller; a successful lookup with no
// addresses returns (false, nil).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't it return (true, nil)?

}
// check if the hostname is resolvable yet; DNS may lag behind the service import
host := psmdb.GetMCSHost(cr, se.Name)
resolved, err := hostResolvable(ctx, r.resolver(), host)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it might be better to append . at the end to have ndots=5 to avoid unnecessary lookups

Comment on lines +153 to +159
// check if the hostname is resolvable yet; DNS may lag behind the service import
host := psmdb.GetMCSHost(cr, se.Name)
resolved, err := hostResolvable(ctx, r.resolver(), host)
if err != nil || !resolved {
log.Info("waiting for imported service hostname to resolve to an IP", "replset", replset.Name, "host", host, "error", err)
return api.AppStateInit, nil, nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if we should do this only in initializing state or cache the results some way, because it feels excessive to perform a lookup on every reconciliation throughout the cluster lifetime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community size/L 100-499 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants