org.apache.skywalking.oap.meter.analyzer.dsl.Expression 89 [grpcServerPool-1-thread-5] ERROR [] - [9.3.0-7ad0406] failed to run

[conghuo]

When upgrading from 9.2.0 to 9.3.0, the following errors occur in k8s monitoring. What causes them?

K8s monitoring data source: otel

Note: it operates normally in 9.2.0

2022-12-08 08:13:52,656 org.apache.skywalking.oap.meter.analyzer.dsl.Expression 89 [grpcServerPool-1-thread-20] ERROR [] - [9.3.0-7ad0406] failed to run "(container_memory_working_set_bytes.tagNotEqual('container' , '').tagNotEqual('pod' , '').retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster' , 'service' , 'pod'])).service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)"
java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: io.kubernetes.client.openapi.ApiException:
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2317) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2283) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950) ~[guava-31.1-jre.jar:?]
	at org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry.findServiceName(K8sInfoRegistry.java:122) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType$1.lambda$execute$0(K8sRetagType.java:40) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(Unknown Source) ~[?:?]
	at org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType$1.execute(K8sRetagType.java:49) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily.retagByK8sMeta(SampleFamily.java:368) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily$retagByK8sMeta$9.call(Unknown Source) ~[?:?]
	at Script1.run(Script1.groovy:1) ~[?:?]
	at org.apache.skywalking.oap.meter.analyzer.dsl.Expression.run(Expression.java:78) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.Analyzer.analyse(Analyzer.java:133) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.MetricConvert.toMeter(MetricConvert.java:93) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.prometheus.PrometheusMetricConverter.toMeter(PrometheusMetricConverter.java:84) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.server.receiver.otel.oc.OCMetricHandler$1.lambda$onNext$6(OCMetricHandler.java:92) ~[otel-receiver-plugin-9.3.0.jar:9.3.0]
	at java.util.ArrayList.forEach(Unknown Source) [?:?]
	at org.apache.skywalking.oap.server.receiver.otel.oc.OCMetricHandler$1.onNext(OCMetricHandler.java:92) [otel-receiver-plugin-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.server.receiver.otel.oc.OCMetricHandler$1.onNext(OCMetricHandler.java:65) [otel-receiver-plugin-9.3.0.jar:9.3.0]
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262) [grpc-stub-1.49.0.jar:1.49.0]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:332) [grpc-core-1.49.0.jar:1.49.0]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:315) [grpc-core-1.49.0.jar:1.49.0]
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834) [grpc-core-1.49.0.jar:1.49.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.49.0.jar:1.49.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.49.0.jar:1.49.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.ExecutionException: io.kubernetes.client.openapi.ApiException:
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2317) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2283) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950) ~[guava-31.1-jre.jar:?]
	at org.apache.skywalking.library.kubernetes.KubernetesPods.findByObjectID(KubernetesPods.java:81) ~[library-kubernetes-support-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry.lambda$new$14(K8sInfoRegistry.java:80) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) ~[guava-31.1-jre.jar:?]
	... 35 more
Caused by: io.kubernetes.client.openapi.ApiException:
	at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:973) ~[client-java-api-16.0.0.jar:?]
	at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:885) ~[client-java-api-16.0.0.jar:?]
	at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedPodWithHttpInfo(CoreV1Api.java:50898) ~[client-java-api-16.0.0.jar:?]
	at io.kubernetes.client.openapi.apis.CoreV1Api.readNamespacedPod(CoreV1Api.java:50874) ~[client-java-api-16.0.0.jar:?]
	at org.apache.skywalking.library.kubernetes.KubernetesPods$2.load(KubernetesPods.java:66) ~[library-kubernetes-support-9.3.0.jar:9.3.0]
	at org.apache.skywalking.library.kubernetes.KubernetesPods$2.load(KubernetesPods.java:61) ~[library-kubernetes-support-9.3.0.jar:9.3.0]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950) ~[guava-31.1-jre.jar:?]
	at org.apache.skywalking.library.kubernetes.KubernetesPods.findByObjectID(KubernetesPods.java:81) ~[library-kubernetes-support-9.3.0.jar:9.3.0]
	at org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry.lambda$new$14(K8sInfoRegistry.java:80) ~[meter-analyzer-9.3.0.jar:9.3.0]
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) ~[guava-31.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) ~[guava-31.1-jre.jar:?]
	... 35 more

[wu-sheng]

otel setup had been changed as well. Please follow 9.3.0 setup doc.

[conghuo]

Configure according to the following document, and upgrade the otel to the latest version 0.67.0

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/skywalkingexporter/README.md

https://skywalking.apache.org/docs/main/v9.3.0/readme/

error:

[wu-sheng]

We don't put any document about skywalking as otlp trace exporter.
Your question doesn't make any chance.

[conghuo]

Solved, thank you for your support!

terms of settlement:

Change opentelemetry exporters to otlp

exporters:

otlp:

endpoint: "skywalking-oap.skywalking.svc.cluster.local:11800"

tls:

insecure: true