Timed out retrying get_from_translog exception in indices:data/read/get[s]

[smalyshev]

Elasticsearch Version

git-5df8a4494b26

Installed Plugins

No response

Java Version

bundled

OS Version

c4a-highcpu-32-lssd

Problem Description

Got a bunch of suppressed REST errors in Serverless all following the same pattern:

org.elasticsearch.transport.RemoteTransportException: [es-es-search-66fdb6dbc6-7tlfs][100.64.49.105:9300][indices:data/read/get[s]]
Caused by: org.elasticsearch.ElasticsearchException: Timed out retrying get_from_translog
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.get.TransportGetAction$1.onTimeout(TransportGetAction.java:258)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:390)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:303)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:682)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: [.kibana_1/6GvgjGnoQ4a9gMB0K7Hzag][[.kibana_1][0]] org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.server@9.1.0/org.elasticsearch.index.shard.IndexShard.lambda$waitForPrimaryTermAndGeneration$56(IndexShard.java:4681)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:394)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:314)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:210)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:175)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.index.shard.IndexShard.waitForEngineOrClosedShard(IndexShard.java:4671)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.index.shard.IndexShard.waitForPrimaryTermAndGeneration(IndexShard.java:4679)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.get.TransportGetAction.lambda$tryGetFromTranslog$7(TransportGetAction.java:296)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1496)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:464)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:424)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)

Steps to Reproduce

?

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[kingherc]

Hi, I did an internal investigation here where basically the errors were because of connectivity issues to the object store. Thus, I believe this is not a bug, and perhaps no further action is needed. Unless we see this happening frequently, but I have no evidence of this at this moment. cc'ing @atyrian with whom I briefly discussed this, to see if it's reasonable to close this at this moment.