Skip to content

Querier panic caught by grpc recovery handler when executing a distributed query #8198

New issue
New issue
Open
#8199
Open
Querier panic caught by grpc recovery handler when executing a distributed query#8198
#8199
Labels
bugcomponent: query
@gburek-fastly

Description

@gburek-fastly
gburek-fastly
opened on Apr 9, 2025

Thanos, Prometheus and Golang version used:
thanos_build_info{branch="HEAD", goarch="amd64", goos="linux", goversion="go1.23.4", tags="netgo", version="0.37.2"}

Object Storage Provider:
GCS

What happened:
A new tiered thanos deployment has been encountering recurrent recovered panics during normal execution on query nodes that are running with --query.mode= distributed . These rescued panics result in grpc_server_started_total - sum without (grpc_code)(grpc_server_handled_total)) for a given query pod increasing without bound, which is a problem due to our use of this query to trigger KEDA auto scaling to handle query pods that have long running queries occupying their queue.

It appears that when a remote query node restarts or the GRPC connection is interrupted, the panicing node, with --query.mode= distributed sees a panic, which is caught by the GRPC handler. Queries will return to the client with a 422.

What you expected to happen:

Query to return without error with partial results.

How to reproduce it (as minimally and precisely as possible):

Appears to happen under load in any query node that is talking to a --query.mode= local node. I'm unable to find a perfect reproduction case.

Full logs to relevant components:

2025-04-09 22:02:21.136   172.18.72.4 - - [09/Apr/2025:22:02:21 +0000]  422 "POST /thanos/api/v1/query HTTP/1.1" 151 "-" "Go-http-client/1.1" "-"

2025-04-09 22:02:20.380	{
  "caller": "grpc.go:77",
  "component": "query",
  "level": "error",
  "msg": "recovered from panic",
  "panic": "runtime error: invalid memory address or nil pointer dereference",
  "service": "gRPC/server",
  "stack": "Z29yb3V0aW5lIDc4NjI2MDEgW3J1bm5pbmddOgpydW50aW1lL2RlYnVnLlN0YWNrKCkKCS91c3IvbG9jYWwvZ28vc3JjL3J1bnRpbWUvZGVidWcvc3RhY2suZ286MjYgKzB4NWUKZ2l0aHViLmNvbS90aGFub3MtaW8vdGhhbm9zL3BrZy9zZXJ2ZXIvZ3JwYy5OZXcuZnVuYzEoezB4MjZjOTdhMCwgMHg1MDVhMjYwfSkKCS9hcHAvcGtnL3NlcnZlci9ncnBjL2dycGMuZ286NzcgKzB4YjQKZ2l0aHViLmNvbS90aGFub3MtaW8vdGhhbm9zL3BrZy9zZXJ2ZXIvZ3JwYy5OZXcuV2l0aFJlY292ZXJ5SGFuZGxlci5mdW5jMTguMSh7MHgwPywgMHgyNjk4OTIwP30sIHsweDI2Yzk3YTA/LCAweDUwNWEyNjA/fSkKCS9nby9wa2cvbW9kL2dpdGh1Yi5jb20vZ3JwYy1lY29zeXN0ZW0vZ28tZ3JwYy1taWRkbGV3YXJlL3YyQHYyLjEuMC9pbnRlcmNlcHRvcnMvcmVjb3Zlcnkvb3B0aW9ucy5nbzozNiArMHgyNwpnaXRodWIuY29tL2dycGMtZWNvc3lzdGVtL2dvLWdycGMtbWlkZGxld2FyZS92Mi9pbnRlcmNlcHRvcnMvcmVjb3ZlcnkucmVjb3ZlckZyb20oezB4MzlkOWYxOD8sIDB4YzAwNjA1ZmIzMD99LCB7MHgyNmM5N2EwPywgMHg1MDVhMjYwP30sIDB4YzAwMGI2NmNjZj8pCgkvZ28vcGtnL21vZC9naXRodWIuY29tL2dycGMtZWNvc3lzdGVtL2dvLWdycGMtbWlkZGxld2FyZS92MkB2Mi4xLjAvaW50ZXJjZXB0b3JzL3JlY292ZXJ5L2ludGVyY2VwdG9ycy5nbzo1NCArMHhlYQpnaXRodWIuY29tL2dycGMtZWNvc3lzdGVtL2dvLWdycGMtbWlkZGxld2FyZS92Mi9pbnRlcmNlcHRvcnMvcmVjb3ZlcnkuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuZnVuYzEuMSgpCgkvZ28vcGtnL21vZC9naXRodWIuY29tL2dycGMtZWNvc3lzdGVtL2dvLWdycGMtbWlkZGxld2FyZS92MkB2Mi4xLjAvaW50ZXJjZXB0b3JzL3JlY292ZXJ5L2ludGVyY2VwdG9ycy5nbzo0NCArMHg3YgpwYW5pYyh7MHgyNmM5N2EwPywgMHg1MDVhMjYwP30pCgkvdXNyL2xvY2FsL2dvL3NyYy9ydW50aW1lL3BhbmljLmdvOjc4NSArMHgxMzIKZ2l0aHViLmNvbS90aGFub3MtaW8vcHJvbXFsLWVuZ2luZS9lbmdpbmUuKCpBbmFseXplT3V0cHV0Tm9kZSkuYWdncmVnYXRlU2FtcGxlcygweDM5YjgxMDA/KQoJL2dvL3BrZy9tb2QvZ2l0aHViLmNvbS90aGFub3MtaW8vcHJvbXFsLWVuZ2luZUB2MC4wLjAtMjAyNDEyMDMxMDMyNDAtMmY0OWY4MGM3YzY4L2VuZ2luZS9leHBsYWluLmdvOjU0ICsweGUKZ2l0aHViLmNvbS90aGFub3MtaW8vcHJvbXFsLWVuZ2luZS9lbmdpbmUuKCpBbmFseXplT3V0cHV0Tm9kZSkuVG90YWxTYW1wbGVzKC4uLikKCS9nby9wa2cvbW9kL2dpdGh1Yi5jb20vdGhhbm9zLWlvL3Byb21xbC1lbmdpbmVAdjAuMC4wLTIwMjQxMjAzMTAzMjQwLTJmNDlmODBjN2M2OC9lbmdpbmUvZXhwbGFpbi5nbzozOQpnaXRodWIuY29tL3RoYW5vcy1pby90aGFub3MvcGtnL2FwaS9xdWVyeS5leHRyYWN0UXVlcnlTdGF0cyh7MHgzOWU0NTgwLCAweGMwMDYwNjZmYzB9KQoJL2FwcC9wa2cvYXBpL3F1ZXJ5L2dycGMuZ286Mjg4ICsweDY1CmdpdGh1Yi5jb20vdGhhbm9zLWlvL3RoYW5vcy9wa2cvYXBpL3F1ZXJ5LigqR1JQQ0FQSSkuUXVlcnkoMHhjMDAwYzQwYWYwLCAweGMwMDE2ODk1MDAsIHsweDM5ZThhMzAsIDB4YzAwNjA1Y2VmMH0pCgkvYXBwL3BrZy9hcGkvcXVlcnkvZ3JwYy5nbzoxNDIgKzB4NzdhCmdpdGh1Yi5jb20vdGhhbm9zLWlvL3RoYW5vcy9wa2cvYXBpL3F1ZXJ5L3F1ZXJ5cGIuX1F1ZXJ5X1F1ZXJ5X0hhbmRsZXIoezB4MjdjNjc4MCwgMHhjMDAwYzQwYWYwfSwgezB4MzllNDRhOCwgMHhjMDA2MDVmZTkwfSkKCS9hcHAvcGtnL2FwaS9xdWVyeS9xdWVyeXBiL3F1ZXJ5LnBiLmdvOjY4OCArMHgxMDcKZ2l0aHViLmNvbS90aGFub3MtaW8vdGhhbm9zL3BrZy9zZXJ2ZXIvZ3JwYy5OZXcuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuZnVuYzIyKHsweDI3YzY3ODAsIDB4YzAwMGM0MGFmMH0sIHsweDM5ZTQ0YTgsIDB4YzAwNjA1ZmU5MH0sIDB4YzAwNjA2MTc0MCwgMHgzNTg5NTI4KQoJL2dvL3BrZy9tb2QvZ2l0aHViLmNvbS9ncnBjLWVjb3N5c3RlbS9nby1ncnBjLW1pZGRsZXdhcmUvdjJAdjIuMS4wL2ludGVyY2VwdG9ycy9zZWxlY3Rvci9zZWxlY3Rvci5nbzo1MiArMHgxZmEKZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwYy5nZXRDaGFpblN0cmVhbUhhbmRsZXIuZnVuYzEoezB4MjdjNjc4MCwgMHhjMDAwYzQwYWYwfSwgezB4MzllNDRhOCwgMHhjMDA2MDVmZTkwfSkKCS9nby9wa2cvbW9kL2dvb2dsZS5nb2xhbmcub3JnL2dycGNAdjEuNjMuMi9zZXJ2ZXIuZ286MTUxNSArMHhiMgpnaXRodWIuY29tL3RoYW5vcy1pby90aGFub3MvcGtnL3NlcnZlci9ncnBjLk5ldy5TdHJlYW1TZXJ2ZXJJbnRlcmNlcHRvci5mdW5jMjAoezB4MjdjNjc4MCwgMHhjMDAwYzQwYWYwfSwgezB4MzllNDRhOCwgMHhjMDA2MDVmZTkwfSwgMHhjMDA2MDYxNzQwLCAweGMwMDYwNjJkNDApCgkvZ28vcGtnL21vZC9naXRodWIuY29tL2dycGMtZWNvc3lzdGVtL2dvLWdycGMtbWlkZGxld2FyZS92MkB2Mi4xLjAvaW50ZXJjZXB0b3JzL3NlbGVjdG9yL3NlbGVjdG9yLmdvOjUyICsweDFmYQpnb29nbGUuZ29sYW5nLm9yZy9ncnBjLmdldENoYWluU3RyZWFtSGFuZGxlci5mdW5jMSh7MHgyN2M2NzgwLCAweGMwMDBjNDBhZjB9LCB7MHgzOWU0NGE4LCAweGMwMDYwNWZlOTB9KQoJL2dvL3BrZy9tb2QvZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwY0B2MS42My4yL3NlcnZlci5nbzoxNTE1ICsweGIyCmdpdGh1Yi5jb20vdGhhbm9zLWlvL3RoYW5vcy9wa2cvdHJhY2luZy90cmFjaW5nX21pZGRsZXdhcmUuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuZnVuYzEoezB4MjdjNjc4MCwgMHhjMDAwYzQwYWYwfSwgezB4MzllMzYxOCwgMHhjMDkyNGNiODYwfSwgMHhjMDA2MDYxNzQwLCAweGMwMDYwNjJiODApCgkvYXBwL3BrZy90cmFjaW5nL2ludGVyY2VwdG9ycy9zZXJ2ZXIuZ286NDAgKzB4MjY0CmdpdGh1Yi5jb20vdGhhbm9zLWlvL3RoYW5vcy9wa2cvdHJhY2luZy5TdHJlYW1TZXJ2ZXJJbnRlcmNlcHRvci5mdW5jMSh7MHgyN2M2NzgwLCAweGMwMDBjNDBhZjB9LCB7MHgzOWUzN2M4LCAweGMwMDYwNWZiNjB9LCAweGMwMDYwNjE3NDAsIDB4YzAwNjA2MmI4MCkKCS9hcHAvcGtnL3RyYWNpbmcvZ3JwYy5nbzo0MiArMHhmNwpnb29nbGUuZ29sYW5nLm9yZy9ncnBjLmdldENoYWluU3RyZWFtSGFuZGxlci5mdW5jMSh7MHgyN2M2NzgwLCAweGMwMDBjNDBhZjB9LCB7MHgzOWUzN2M4LCAweGMwMDYwNWZiNjB9KQoJL2dvL3BrZy9tb2QvZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwY0B2MS42My4yL3NlcnZlci5nbzoxNTE1ICsweGIyCmdpdGh1Yi5jb20vdGhhbm9zLWlvL3RoYW5vcy9wa2cvc2VydmVyL2dycGMuTmV3LigqU2VydmVyTWV0cmljcykuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuU3RyZWFtU2VydmVySW50ZXJjZXB0b3IuZnVuYzI2KHsweDI3YzY3ODAsIDB4YzAwMGM0MGFmMH0sIHsweDM5ZTM5MDgsIDB4YzA5MjRjYjg0MH0sIDB4YzAwNjA2MTc0MD8sIDB4YzAwNjA2MmI0MCkKCS9nby9wa2cvbW9kL2dpdGh1Yi5jb20vZ3JwYy1lY29zeXN0ZW0vZ28tZ3JwYy1taWRkbGV3YXJlL3YyQHYyLjEuMC9pbnRlcmNlcHRvcnMvc2VydmVyLmdvOjM1ICsweDJiYgpnb29nbGUuZ29sYW5nLm9yZy9ncnBjLmdldENoYWluU3RyZWFtSGFuZGxlci5mdW5jMSh7MHgyN2M2NzgwLCAweGMwMDBjNDBhZjB9LCB7MHgzOWUzOTA4LCAweGMwOTI0Y2I4NDB9KQoJL2dvL3BrZy9tb2QvZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwY0B2MS42My4yL3NlcnZlci5nbzoxNTE1ICsweGIyCmdpdGh1Yi5jb20vZ3JwYy1lY29zeXN0ZW0vZ28tZ3JwYy1taWRkbGV3YXJlL3YyL2ludGVyY2VwdG9ycy9yZWNvdmVyeS5TdHJlYW1TZXJ2ZXJJbnRlcmNlcHRvci5mdW5jMSh7MHgyN2M2NzgwPywgMHhjMDAwYzQwYWYwP30sIHsweDM5ZTM5MDg/LCAweGMwOTI0Y2I4NDA/fSwgMHhjMDA2MDYxNzQwPywgMHgzNTg5NTI4PykKCS9nby9wa2cvbW9kL2dpdGh1Yi5jb20vZ3JwYy1lY29zeXN0ZW0vZ28tZ3JwYy1taWRkbGV3YXJlL3YyQHYyLjEuMC9pbnRlcmNlcHRvcnMvcmVjb3ZlcnkvaW50ZXJjZXB0b3JzLmdvOjQ4ICsweDgyCmdvb2dsZS5nb2xhbmcub3JnL2dycGMuZ2V0Q2hhaW5TdHJlYW1IYW5kbGVyLmZ1bmMxKHsweDI3YzY3ODAsIDB4YzAwMGM0MGFmMH0sIHsweDM5ZTM5MDgsIDB4YzA5MjRjYjg0MH0pCgkvZ28vcGtnL21vZC9nb29nbGUuZ29sYW5nLm9yZy9ncnBjQHYxLjYzLjIvc2VydmVyLmdvOjE1MTUgKzB4YjIKZ2l0aHViLmNvbS90aGFub3MtaW8vdGhhbm9zL3BrZy9zZXJ2ZXIvZ3JwYy5OZXcuTmV3U3RyZWFtU2VydmVyUmVxdWVzdElESW50ZXJjZXB0b3IuZnVuYzE3KHsweDI3YzY3ODAsIDB4YzAwMGM0MGFmMH0sIHsweDM5ZTM3MzgsIDB4YzAwMDk4ZDRhMH0sIDB4YzAwNjA2MTc0MD8sIDB4YzAwNjA2MmFjMCkKCS9hcHAvcGtnL3NlcnZlci9ncnBjL3JlcXVlc3RfaWQuZ286NTAgKzB4YTYKZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwYy5OZXdTZXJ2ZXIuY2hhaW5TdHJlYW1TZXJ2ZXJJbnRlcmNlcHRvcnMuY2hhaW5TdHJlYW1JbnRlcmNlcHRvcnMuZnVuYzIoezB4MjdjNjc4MCwgMHhjMDAwYzQwYWYwfSwgezB4MzllMzczOCwgMHhjMDAwOThkNGEwfSwgMHhjMDA2MDYxNzQwLCAweGMwMDYwNWNkMjA/KQoJL2dvL3BrZy9tb2QvZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwY0B2MS42My4yL3NlcnZlci5nbzoxNTA2ICsweDg1Cmdvb2dsZS5nb2xhbmcub3JnL2dycGMuKCpTZXJ2ZXIpLnByb2Nlc3NTdHJlYW1pbmdSUEMoMHhjMDAwZDJiYzAwLCB7MHgzOWQ5ZjE4LCAweGMwMDYwNWZhMTB9LCB7MHgzOWYxZWEwLCAweGMwMjcxMmE0ODB9LCAweGMwMDBjNzZjNjAsIDB4YzAwMDk5MDYzMCwgMHg1MDhjYzIwLCAweDApCgkvZ28vcGtnL21vZC9nb29nbGUuZ29sYW5nLm9yZy9ncnBjQHYxLjYzLjIvc2VydmVyLmdvOjE2NzAgKzB4MTFlNwpnb29nbGUuZ29sYW5nLm9yZy9ncnBjLigqU2VydmVyKS5oYW5kbGVTdHJlYW0oMHhjMDAwZDJiYzAwLCB7MHgzOWYxZWEwLCAweGMwMjcxMmE0ODB9LCAweGMwMDBjNzZjNjApCgkvZ28vcGtnL21vZC9nb29nbGUuZ29sYW5nLm9yZy9ncnBjQHYxLjYzLjIvc2VydmVyLmdvOjE3ODQgKzB4ZTNhCmdvb2dsZS5nb2xhbmcub3JnL2dycGMuKCpTZXJ2ZXIpLnNlcnZlU3RyZWFtcy5mdW5jMi4xKCkKCS9nby9wa2cvbW9kL2dvb2dsZS5nb2xhbmcub3JnL2dycGNAdjEuNjMuMi9zZXJ2ZXIuZ286MTAxOSArMHg3ZgpjcmVhdGVkIGJ5IGdvb2dsZS5nb2xhbmcub3JnL2dycGMuKCpTZXJ2ZXIpLnNlcnZlU3RyZWFtcy5mdW5jMiBpbiBnb3JvdXRpbmUgNzgxMTU2NwoJL2dvL3BrZy9tb2QvZ29vZ2xlLmdvbGFuZy5vcmcvZ3JwY0B2MS42My4yL3NlcnZlci5nbzoxMDMwICsweDEyNQo=",
  "ts": "2025-04-09T04:07:56.772695341Z"
}

decoded from base64 stack:

goroutine 144567308 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/thanos-io/thanos/pkg/server/grpc.New.func1({0x26c97a0, 0x505a260})
	/app/pkg/server/grpc/grpc.go:77 +0xb4
github.com/thanos-io/thanos/pkg/server/grpc.New.WithRecoveryHandler.func18.1({0x0?, 0x2698920?}, {0x26c97a0?, 0x505a260?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/recovery/options.go:36 +0x27
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x39d9f18?, 0xc014d85cb0?}, {0x26c97a0?, 0x505a260?}, 0xc004b64ccf?)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/recovery/interceptors.go:54 +0xea
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1.1()
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/recovery/interceptors.go:44 +0x7b
panic({0x26c97a0?, 0x505a260?})
	/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/thanos-io/promql-engine/engine.(*AnalyzeOutputNode).aggregateSamples(0x39b8100?)
	/go/pkg/mod/github.com/thanos-io/[email protected]/engine/explain.go:54 +0xe
github.com/thanos-io/promql-engine/engine.(*AnalyzeOutputNode).TotalSamples(...)
	/go/pkg/mod/github.com/thanos-io/[email protected]/engine/explain.go:39
github.com/thanos-io/thanos/pkg/api/query.extractQueryStats({0x39e4580, 0xc014d89260})
	/app/pkg/api/query/grpc.go:288 +0x65
github.com/thanos-io/thanos/pkg/api/query.(*GRPCAPI).Query(0xc000b1a910, 0xc014d13f00, {0x39e8a30, 0xc014d870d0})
	/app/pkg/api/query/grpc.go:142 +0x77a
github.com/thanos-io/thanos/pkg/api/query/querypb._Query_Query_Handler({0x27c6780, 0xc000b1a910}, {0x39e44a8, 0xc014daa090})
	/app/pkg/api/query/querypb/query.pb.go:688 +0x107
github.com/thanos-io/thanos/pkg/server/grpc.New.StreamServerInterceptor.func22({0x27c6780, 0xc000b1a910}, {0x39e44a8, 0xc014daa090}, 0xc014d96660, 0x3589528)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/selector/selector.go:52 +0x1fa
google.golang.org/grpc.getChainStreamHandler.func1({0x27c6780, 0xc000b1a910}, {0x39e44a8, 0xc014daa090})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1515 +0xb2
github.com/thanos-io/thanos/pkg/server/grpc.New.StreamServerInterceptor.func20({0x27c6780, 0xc000b1a910}, {0x39e44a8, 0xc014daa090}, 0xc014d96660, 0xc014da0540)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/selector/selector.go:52 +0x1fa
google.golang.org/grpc.getChainStreamHandler.func1({0x27c6780, 0xc000b1a910}, {0x39e44a8, 0xc014daa090})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1515 +0xb2
github.com/thanos-io/thanos/pkg/tracing/tracing_middleware.StreamServerInterceptor.StreamServerInterceptor.func1({0x27c6780, 0xc000b1a910}, {0x39e3618, 0xc08b368680}, 0xc014d96660, 0xc014da0380)
	/app/pkg/tracing/interceptors/server.go:40 +0x264
github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1({0x27c6780, 0xc000b1a910}, {0x39e37c8, 0xc014d85ce0}, 0xc014d96660, 0xc014da0380)
	/app/pkg/tracing/grpc.go:42 +0xf7
google.golang.org/grpc.getChainStreamHandler.func1({0x27c6780, 0xc000b1a910}, {0x39e37c8, 0xc014d85ce0})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1515 +0xb2
github.com/thanos-io/thanos/pkg/server/grpc.New.(*ServerMetrics).StreamServerInterceptor.StreamServerInterceptor.func26({0x27c6780, 0xc000b1a910}, {0x39e3908, 0xc08b368660}, 0xc014d96660?, 0xc014da0340)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x27c6780, 0xc000b1a910}, {0x39e3908, 0xc08b368660})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1515 +0xb2
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1({0x27c6780?, 0xc000b1a910?}, {0x39e3908?, 0xc08b368660?}, 0xc014d96660?, 0x3589528?)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/recovery/interceptors.go:48 +0x82
google.golang.org/grpc.getChainStreamHandler.func1({0x27c6780, 0xc000b1a910}, {0x39e3908, 0xc08b368660})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1515 +0xb2
github.com/thanos-io/thanos/pkg/server/grpc.New.NewStreamServerRequestIDInterceptor.func17({0x27c6780, 0xc000b1a910}, {0x39e3738, 0xc09466c690}, 0xc014d96660?, 0xc014da02c0)
	/app/pkg/server/grpc/request_id.go:50 +0xa6
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x27c6780, 0xc000b1a910}, {0x39e3738, 0xc09466c690}, 0xc014d96660, 0xc014d86f00?)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1506 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000d43c00, {0x39d9f18, 0xc014d85b90}, {0x39f1ea0, 0xc01f800f00}, 0xc014d22360, 0xc000aafc20, 0x508cc20, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1670 +0x11e7
google.golang.org/grpc.(*Server).handleStream(0xc000d43c00, {0x39f1ea0, 0xc01f800f00}, 0xc014d22360)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1784 +0xe3a
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1019 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 143974538
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1030 +0x125

Anything else we need to know:

query-sidecars:

args:
  - "query"
  - "--grpc-address=0.0.0.0:10901"
  - "--http-address=0.0.0.0:10902"
  - |
    --tracing.config=
      config:
        client_type: http
        compression: gzip
        endpoint: $(HOST_IP):4318
        insecure: true
        sampler_type: parentbasedneversample
      type: OTLP
  - "--query.max-concurrent=20"
  - "--endpoint-group=dns+thanos-api-query-sidecar-shards-0-headless.svc:10901"
  - "--endpoint-group=dns+thanos-api-query-sidecar-shards-1-headless.svc:10901"
  - "--endpoint-group=dns+thanos-api-query-sidecar-shards-2-headless.svc:10901"
  - "--endpoint-group=dns+thanos-api-query-sidecar-shards-3-headless.svc:10901"
  - "--endpoint-group=dns+thanos-api-query-sidecar-shards-4-headless.svc:10901"
  - "--http-grace-period=1m"
  - "--log.format=json"
  - "--enable-auto-gomemlimit"
  - "--grpc-compression=snappy"
  - "--grpc-grace-period=1m"
  - "--grpc-server-max-connection-age=1m"
  - "--query.default-step=15s"
  - "--query.max-concurrent-select=4"
  - "--query.mode=distributed"
  - "--query.promql-engine=thanos"
  - "--query.replica-label=replica"
  - "--query.timeout=2m"
  - "--store.sd-dns-interval=1s"
  - "--query.auto-downsampling"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugcomponent: query

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions