[SecurityAdmin] unexpected SocketTimeoutException

[amirkkn]

Hi,
on one of our cluster we get socket timeout exceptions suddenly during securityAdmin script execution. We run opensearch 2.19.1 on kubernetes and in other clusters with the same version works as expected. I tried the -diagnose flag but I didn't see any error or something unusual. The cluster is GREEN and it seems functional. Could you please guide me how could I resolve this issue?

BTW, Is there a way to increase the timeout?

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=admin"
OpenSearch Version: 2.19.1
Contacting opensearch cluster 'opensearch' ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 11
Number of data nodes: 5
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
Will update '/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/allowlist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
ERR: An unexpected SocketTimeoutException occured: 30,000 milliseconds timeout on connection http-outgoing-15 [ACTIVE]
Trace:
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-15 [ACTIVE]
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1230)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:358)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:346)
	at org.opensearch.security.tools.SecurityAdmin.upload(SecurityAdmin.java:1516)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:978)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:165)
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-15 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:98)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:40)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:506)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:1583)


command terminated with exit code 255

Thanks

[cwperks]

[Triage] @amirkkn thank you for filing this issue. I see that the messages indicate that individual updates were successful. Did the script still work as expected even with the error in the output?

Any other info you can provide like how many entries are in each yaml file, potentially the (redacted) contents of the yaml file and number of nodes in the cluster? (Number of nodes + data nodes is in the script output posted in OP)

I've never seen this error before.

[cwperks]

Potentially related: opensearch-project/OpenSearch#19458

[amirkkn]

[Triage] @amirkkn thank you for filing this issue. I see that the messages indicate that individual updates were successful. Did the script still work as expected even with the error in the output?

Any other info you can provide like how many entries are in each yaml file, potentially the (redacted) contents of the yaml file and number of nodes in the cluster? (Number of nodes + data nodes is in the script output posted in OP)

I've never seen this error before.

it is a good point, thanks. I have on this environment for roles.yml around 460 entries and in other working environments maximal 200. Could it be related as you indicated? Is there any limitation or could we increase the timout somehow from 30s to 60s? I couldn't find any working solution for increasing it
Thanks in advance

[nibix]

@amirkkn can you please check the logs on the cluster nodes while you get the timeout issue?

It is possbile that this issue is caused by an error on one of the nodes.

[amirkkn]

@amirkkn can you please check the logs on the cluster nodes while you get the timeout issue?

It is possbile that this issue is caused by an error on one of the nodes.

No, no error logs on nodes found

[amirkkn]

[Triage] @amirkkn thank you for filing this issue. I see that the messages indicate that individual updates were successful. Did the script still work as expected even with the error in the output?

Any other info you can provide like how many entries are in each yaml file, potentially the (redacted) contents of the yaml file and number of nodes in the cluster? (Number of nodes + data nodes is in the script output posted in OP)

I've never seen this error before.

Unfortunately is the issue still there after optimising the roles and role_mapping yamls. I'm wondering why securityAdmin cli does not support timeout configuration!

Anyway. We have 5 data nodes, 3 masters, 3 coordinating. It seems the execution was successful nevertheless. I'm expecting something like that for final execution steps :

11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
11:57:11  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null

If issue related to opensearch-project/OpenSearch#19458, then an upgrade might be necessary.

[amirkkn]

[Triage] @amirkkn thank you for filing this issue. I see that the messages indicate that individual updates were successful. Did the script still work as expected even with the error in the output?

Any other info you can provide like how many entries are in each yaml file, potentially the (redacted) contents of the yaml file and number of nodes in the cluster? (Number of nodes + data nodes is in the script output posted in OP)

I've never seen this error before.

I have optimized the yamls and reduced a lot. The issue still exists. I'm wondering why securityAdmin script does not provide timeout param for customising. It seems the