mongodb-healthcheck readiness logs repeated saslSupportedMechs ProtocolError because AuthSource is unset

[jmbravo]

Report

The default mongodb-healthcheck readiness probe in Operator 1.22.0 generates an informational ProtocolError in the mongod log every three seconds when authenticating as the clusterMonitor system user.

The probe succeeds and the Pod remains Ready, but the repeated messages look like authentication failures and create significant log noise:

{"s":"I","c":"ACCESS","id":5286202,"msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication from clusterMonitor@ to @admin"}}}

The same connection immediately authenticates successfully:

{"s":"I","c":"ACCESS","id":5286306,"msg":"Successfully authenticated","attr":{"client":"127.0.0.1:<port>","mechanism":"SCRAM-SHA-256","user":"clusterMonitor","db":"admin","result":0}}

There are no real authentication failures, the readiness probe succeeds, and the clusterMonitor user exists in the admin database with the expected roles.

Steps to reproduce

1. Deploy Percona Operator for MongoDB 1.22.0 with authentication enabled.

2. Use the default mongod readiness probe:

   /opt/percona/mongodb-healthcheck k8s readiness --component mongod
   

3. Ensure the standard clusterMonitor user exists in admin.

4. Inspect the mongod logs.

5. Observe the saslSupportedMechs ProtocolError every periodSeconds: 3, followed by successful authentication on the same connection.

Expected behavior

The default readiness healthcheck should authenticate as clusterMonitor@admin without first requesting SASL mechanisms for clusterMonitor@ with an empty authentication source, and should not produce a ProtocolError on every probe.

Actual behavior

The probe requests supported SASL mechanisms with an empty database, then authenticates against admin. Authentication succeeds, but MongoDB logs a misleading ProtocolError every time the readiness probe runs.

Suspected cause

In tag v1.22.0, pkg/psmdb/mongo/mongo.go builds the Go driver credential with only Username and Password:

opts.SetAuth(options.Credential{
    Password: conf.Password,
    Username: conf.Username,
})

mongo.Config has no AuthSource field in this release, and cmd/mongodb-healthcheck/tool/config.go therefore cannot explicitly set the healthcheck authentication source to admin.

The current main branch has added AuthSource to mongo.Config and passes it into options.Credential, but cmd/mongodb-healthcheck/tool.NewConfig still appears not to populate it. The behavior should therefore be covered by a regression test, even if the Go driver v2 migration happens to apply a default differently.

Suggested fix

Set the authentication source explicitly for the healthcheck system user:

conf.AuthSource = "admin"

and ensure it is passed to the driver credential:

opts.SetAuth(options.Credential{
    Username:   conf.Username,
    Password:   conf.Password,
    AuthSource: conf.AuthSource,
})

Please also add a test asserting that the healthcheck credential uses admin as its authentication source.

Versions

• Percona Operator for MongoDB: 1.22.0
• Percona Server for MongoDB: 8.0.19-7
• Readiness healthcheck commit reported by the binary: 1fc1e8b4e5640f83f0b5b71e0f25d8bbec82caf6 (release-1-22-0)
• Authentication mechanism selected successfully: SCRAM-SHA-256

Additional context

Replacing the readiness probe with a TCP socket check removes the log noise, but weakens the semantic check. The desired behavior is for the default authenticated readiness probe to work without producing misleading protocol errors.

[egegunes]

@jmbravo thanks for the report. I created https://perconadev.atlassian.net/browse/K8SPSMDB-1764 to fix this.