A few sporadic warnings and errors were found in the logs of the proxy and querynode nodes, appearing intermittently. #49227
Unanswered
bozai111
asked this question in
Q&A and General discussion
Replies: 2 comments
-
|
I think it is expected behavior in a busy cluster, the warning/error is harmless:
What's happening:
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
A few sporadic warnings and errors were found in the logs of the proxy and querynode nodes, appearing intermittently.
milvus cluster version :2.5.13
External Pulsar and object storage used
Managed using the milvus operator
-- querynode log
loadSegmentFunc,error:At LoadDeltaLogs: context canceled,errorVerbose:At LoadDeltaLogs: context canceled
(1) attached stack trace
-- stack trace:
| github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:357
| github.com/milvus-io/milvus/pkg/v2/util/funcutil.ProcessFuncParallel.func3
| /go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:87
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1700
Wraps: (2) At LoadDeltaLogs
Wraps: (3) context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString,idx:0,stack:github.com/milvus-io/milvus/pkg/v2/util/funcutil.ProcessFuncParallel.func3
/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:89
release new segment created due to load failure,segmentID:460738764747203101,error:At LoadDeltaLogs: context canceled,errorVerbose:At LoadDeltaLogs: context canceled
(1) attached stack trace
-- stack trace:
| github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:357
| github.com/milvus-io/milvus/pkg/v2/util/funcutil.ProcessFuncParallel.func3
| /go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:87
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1700
Wraps: (2) At LoadDeltaLogs
Wraps: (3) context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString
delegator failed to load segments,error:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:575 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:589 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:106 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:167 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments
/go/src/github.com/milvus-io/milvus/internal/querynodev2/cluster/worker.go:103 github.com/milvus-io/milvus/internal/querynodev2/cluster.(*remoteWorker).LoadSegments
/go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator_data.go:437 github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).LoadSegments.(*shardDelegator).LoadSegments.func2.func7
/go/src/github.com/milvus-io/milvus/pkg/util/conc/singleflight.go:19 github.com/milvus-io/milvus/pkg/v2/util/conc.(*Singleflight[...]).Do.func1
/go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:198 golang.org/x/sync/singleflight.(*Group).doCall.func2
/go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:200 golang.org/x/sync/singleflight.(*Group).doCall: rpc error: code = Canceled desc = context canceled,errorVerbose:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace: rpc error: code = Canceled desc = context canceled
(1) attached stack trace
-- stack trace:
| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:575
| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:589
| github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
| /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:106
| github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:167
| github.com/milvus-io/milvus/internal/querynodev2/cluster.(*remoteWorker).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/cluster/worker.go:103
| github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).LoadSegments.(*shardDelegator).LoadSegments.func2.func7
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator_data.go:437
| github.com/milvus-io/milvus/pkg/v2/util/conc.(*Singleflight[...]).Do.func1
| /go/src/github.com/milvus-io/milvus/pkg/util/conc/singleflight.go:19
| golang.org/x/sync/singleflight.(*Group).doCall.func2
| /go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:198
| golang.org/x/sync/singleflight.(*Group).doCall
| /go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:200
| golang.org/x/sync/singleflight.(*Group).Do
| /go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:113
| github.com/milvus-io/milvus/pkg/v2/util/conc.(*Singleflight[...]).Do
| /go/src/github.com/milvus-io/milvus/pkg/util/conc/singleflight.go:18
| github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).LoadSegments.func2
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator_data.go:436
| github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator_data.go:461
| github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/services.go:473
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:306
| github.com/milvus-io/milvus/pkg/v2/proto/querypb._QueryNode_LoadSegments_Handler.func1
| /go/src/github.com/milvus-io/milvus/pkg/proto/querypb/query_coord_grpc.pb.go:2182
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ServerIDValidationUnaryServerInterceptor.func7
| /go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1
| /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ClusterValidationUnaryServerInterceptor.func6
| /go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1
| /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
| github.com/milvus-io/milvus/pkg/v2/util/logutil.UnaryTraceLoggerInterceptor
| /go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:26
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1
| /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
| github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8
| /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
| github.com/milvus-io/milvus/pkg/v2/proto/querypb._QueryNode_LoadSegments_Handler
| /go/src/github.com/milvus-io/milvus/pkg/proto/querypb/query_coord_grpc.pb.go:2184
| google.golang.org/grpc.(*Server).processUnaryRPC
| /go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1379
| google.golang.org/grpc.(*Server).handleStream
| /go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1790
| google.golang.org/grpc.(*Server).serveStreams.func2.1
| /go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1700
Wraps: (2) stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace
| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:575 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:589 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
| /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:106 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
| /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:167 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/cluster/worker.go:103 github.com/milvus-io/milvus/internal/querynodev2/cluster.(*remoteWorker).LoadSegments
| /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator_data.go:437 github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).LoadSegments.(*shardDelegator).LoadSegments.func2.func7
| /go/src/github.com/milvus-io/milvus/pkg/util/conc/singleflight.go:19 github.com/milvus-io/milvus/pkg/v2/util/conc.(*Singleflight[...]).Do.func1
| /go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:198 golang.org/x/sync/singleflight.(*Group).doCall.func2
| /go/pkg/mod/golang.org/x/sync@v0.12.0/singleflight/singleflight.go:200 golang.org/x/sync/singleflight.(*Group).doCall
Wraps: (3) rpc error: code = Canceled desc = context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error
-- proxy log
retry func failed, ctx done,retried:0,attempt:2
fail to execute complex delete,deleteCnt:0,interval:353.263271ms,error:failed to search/query delegator 14 for channel project-mv-tianji-rootcoord-dml_8_460738762474985980v0: rpc error: code = Canceled desc = context canceled,errorVerbose:failed to search/query delegator 14 for channel project-mv-tianji-rootcoord-dml_8_460738762474985980v0: rpc error: code = Canceled desc = context canceled
(1) attached stack trace
-- stack trace:
| github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1
| /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:220
| github.com/milvus-io/milvus/pkg/v2/util/retry.Do
| /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44
| github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry
| /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:181
| github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1
| /go/src/github.com/milvus-io/milvus/internal/proxy/lb_policy.go:248
| golang.org/x/sync/errgroup.(*Group).Go.func1
| /go/pkg/mod/golang.org/x/sync@v0.12.0/errgroup/errgroup.go:78
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1700
Wraps: (2) failed to search/query delegator 14 for channel project-mv-tianji-rootcoord-dml_8_460738762474985980v0
Wraps: (3) rpc error: code = Canceled desc = context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error
Failed to run delete task: failed to search/query delegator 14 for channel project-mv-tianji-rootcoord-dml_8_460738762474985980v0: rpc error: code = Canceled desc = context canceled,stack:github.com/milvus-io/milvus/internal/proxy.(*Proxy).Delete
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2791
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Delete
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:796
github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Delete_Handler.func1
/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.5.13/milvuspb/milvus_grpc.pb.go:2096
github.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor
/go/src/github.com/milvus-io/milvus/internal/proxy/connection/util.go:61
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor
/go/src/github.com/milvus-io/milvus/internal/proxy/trace_log_interceptor.go:36
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor
/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:47
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.RateLimitInterceptor.func5
/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:71
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/pkg/v2/util/logutil.UnaryTraceLoggerInterceptor
/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:26
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func4
/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:112
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/proxy.HookInterceptor
/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:56
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerHookInterceptor.func3
/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:21
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.DatabaseInterceptor.func2
/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.GrpcAuthInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proxy/authentication_interceptor.go:64
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor
/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:38
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Delete_Handler
/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.5.13/milvuspb/milvus_grpc.pb.go:2098
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1379
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1790
google.golang.org/grpc.(*Server).serveStreams.func2.1
/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029
segmentID:460738764747203101 is currently showing as dropped.
Beta Was this translation helpful? Give feedback.
All reactions