Can't split instance id into 2 parts, .1_ #8325

solarzc · 2021-12-22T11:31:51Z

solarzc
Dec 22, 2021

oap version:8.6 / 8.8 /8.+

when start in k8s deployment , get wrong log ,Im sure have cleaned elasticsearch :

2021-12-22 11:13:07,936 - org.apache.skywalking.oap.server.library.server.grpc.GRPCServer - 140 [main] INFO  [] - Bind handler MeterServiceHandlerCompat into gRPC server 0.0.0.0:11800
2021-12-22 11:13:08,243 - org.apache.skywalking.oap.server.library.server.jetty.JettyServer - 107 [main] INFO  [] - start server, host: 0.0.0.0, port: 12800
2021-12-22 11:13:08,246 - org.eclipse.jetty.server.Server - 375 [main] INFO  [] - jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_111-8u111-b14-2~bpo8+1-b14
2021-12-22 11:13:08,280 - org.eclipse.jetty.server.handler.ContextHandler - 916 [main] INFO  [] - Started o.e.j.s.ServletContextHandler@55849272{/,null,AVAILABLE}
2021-12-22 11:13:08,293 - org.eclipse.jetty.server.AbstractConnector - 331 [main] INFO  [] - **Started ServerConnector@396e6d9{HTTP/1.1, (http/1.1)}{0.0.0.0:12800}**
2021-12-22 11:13:08,294 - org.eclipse.jetty.server.Server - 415 [main] INFO  [] - Started @11056ms
2021-12-22 11:13:08,295 - org.apache.skywalking.oap.server.core.storage.PersistenceTimer - 62 [main] INFO  [] - persistence timer start
2021-12-22 11:13:08,299 - org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer - 46 [main] INFO  [] - Cache updateServiceInventory timer start
2021-12-22 11:13:40,328 - org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker - 196 [pool-6-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split instance id into 2 parts, .1_
	at org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:83) ~[server-alarm-plugin-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:203) ~[server-core-8.6.0.jar:8.6.0]
	at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_111]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:203) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:186) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:142) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:115) ~[server-core-8.6.0.jar:8.6.0]
	at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_111]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:110) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:86) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.6.0.jar:8.6.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2021-12-22 11:13:40,337 - org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker - 196 [pool-6-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split instance id into 2 parts, .1_
	at org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:83) ~[server-alarm-plugin-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:203) ~[server-core-8.6.0.jar:8.6.0]
	at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_111]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:203) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:186) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:142) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:115) ~[server-core-8.6.0.jar:8.6.0]
	at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_111]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:110) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:86) ~[server-core-8.6.0.jar:8.6.0]
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.6.0.jar:8.6.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oap
  namespace: skywalking
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oap
  template:
    metadata:
      labels:
        app: oap
        release: skywalking
    spec:
#      nodeName: "ip-10-221-20-74.cn-northwest-1.compute.internal"
      containers:
      - name: oap
        image: apache/skywalking-oap-server:8.6.0-es6
        #image: 10.0.7.238/ops/sky:8.6
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 11800
          name: grpc
        - containerPort: 12800
          name: rest
        resources:
          requests:
            memory: 2Gi
          limits:
            memory: 3Gi
        env:
        - name: JAVA_OPTS
          value: "-Xmx2g -Xms2g"
        - name: SW_CLUSTER
          value: standalone
        - name: SKYWALKING_COLLECTOR_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: SW_STORAGE
          value: elasticsearch
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: vpc-cmp-uat*****.com.cn:443
        - name: SW_NAMESPACE
          value: skywalking
        - name: SW_STORAGE_ES_HTTP_PROTOCOL
          value: https
        - name: SW_RECEIVER_SHARING_GRPC_THREAD_POOL_SIZE
          value: "40"
        - name: SW_RECEIVER_SHARING_GRPC_THREAD_POOL_QUEUE_SIZE
          value: "20000"

wu-sheng · 2021-12-22T12:46:11Z

wu-sheng
Dec 22, 2021
Collaborator

It seems in some cases, service name and instance name are both empty?

2 replies

solarzc Dec 23, 2021
Author

it seems not the same problem.

wu-sheng Dec 23, 2021
Collaborator

This is k8s endpoint. My point is SkyWalking's endpoint concept. Read the following reply, please.

wu-sheng · 2021-12-23T00:58:30Z

wu-sheng
Dec 23, 2021
Collaborator

We fixed one, but just in service mesh TCP case. #8327

1 reply

wu-sheng Dec 23, 2021
Collaborator

Also, in the same theory, if in some case, the endpoint name is null/empty, this error should show up.

MVC4 · 2023-03-22T10:28:22Z

MVC4
Mar 22, 2023

slow sql option ,when choice "View Related Traces", it pop up Exception while fetching data (/endpoint) : Can't split endpoint id into 2 parts.
env:
skywalking version: 9.3.0
-Dskywalking.agent.service_name=xxxxxx
-Dskywalking.agent.instance_name=xxxxxxxxxxx

1 reply

wu-sheng Mar 24, 2023
Collaborator

This is not relative to this thread. Please read more clearly when you reply.
This has been fixed through apache/skywalking-booster-ui#247

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Can't split instance id into 2 parts, .1_ #8325

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Can't split instance id into 2 parts, .1_ #8325

Uh oh!

solarzc Dec 22, 2021

Replies: 3 comments · 4 replies

Uh oh!

Uh oh!

wu-sheng Dec 22, 2021 Collaborator

Uh oh!

solarzc Dec 23, 2021 Author

Uh oh!

wu-sheng Dec 23, 2021 Collaborator

Uh oh!

wu-sheng Dec 23, 2021 Collaborator

Uh oh!

wu-sheng Dec 23, 2021 Collaborator

Uh oh!

MVC4 Mar 22, 2023

Uh oh!

wu-sheng Mar 24, 2023 Collaborator

solarzc
Dec 22, 2021

Replies: 3 comments 4 replies

wu-sheng
Dec 22, 2021
Collaborator

solarzc Dec 23, 2021
Author

wu-sheng Dec 23, 2021
Collaborator

wu-sheng
Dec 23, 2021
Collaborator

wu-sheng Dec 23, 2021
Collaborator

MVC4
Mar 22, 2023

wu-sheng Mar 24, 2023
Collaborator