Mystery NPE on search Cannot invoke \"java.lang.Long.longValue()\" because \"value\" is null

[benwtrent]

Elasticsearch Version

current

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

Have seen a mystery NPE in some server logs. Haven't been able to fully indicate the cause. Likely we need to first improve the "guess root cause" logic & logging.

Steps to Reproduce

N/A

Logs (if relevant)

 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:638) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:323) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:683) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:422) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:276) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:593) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1776) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1509) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1643) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1618) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:45) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.search.SearchService.lambda$maybeWrapListenerForStackTrace$2(SearchService.java:559) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:152) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) 
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 
 at java.base/java.lang.Thread.run(Thread.java:1575) 
Caused by: org.elasticsearch.ElasticsearchException$1: Cannot invoke \"java.lang.Long.longValue()\" because \"value\" is null 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:725) 
 at org.elasticsearch.server@9.1.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:321) 
 ... 26 more 
Caused by: java.lang.NullPointerException: Cannot invoke \"java.lang.Long.longValue()\" because \"value\" is null 

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[benwtrent]

Well, doing a search for value.longValue() doesn't come up with much that's easily determinable within the _search API path for a given shard.

I don't really see anything for value.longValue() in Lucene or Elasticsearch that is an obvious cause :/

There was some ql stuff, but, eh, when I saw this exception it was against a _search API.

[thecoop]

This can also be triggered by implicit unboxing - Long value = null; long l = value;

[javanna]

I have been chasing unexpected errors too recently and I think that the lack of a proper stacktrace here is a serious issue. We avoiding piling up exceptions to send to the coord node when they are not requested, but we lose them in logs too which makes bugs difficult to chase down.

@piergm can you remind me if this is expected? I was hoping that we would get these logs from the data node, with the stacktrace, is that the case? It looks like here we log it in the coord node, which is not that useful is it?

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)