[Bug] graphql queryInstances response empty pods

[mattwmj]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

Today we found that service instances table is empty. It showed instances properly days ago.
The response of graphql queryInstances returns empty pods.
Other metrics, including Endpoint, Topology and Trace, have data.

Since all the service instances are working, it does not make sense that the instances table is empty.
And we did not find any error logs in oap server log or agent log.

Please advice.

FYI

Request:

{
    "query": "query queryInstances($serviceId: ID!, $duration: Duration!) {\n  pods: listInstances(duration: $duration, serviceId: $serviceId) {\n    id\n    value: name\n    label: name\n    language\n    instanceUUID\n    attributes {\n      name\n      value\n    }\n  }\n  }",
    "variables": {
        "serviceId": "cG1zLXdlYi1zZXJ2aWNlcw==.1",
        "duration": {
            "start": "2021-02-08 1629",
            "end": "2024-03-08 1659",
            "step": "MINUTE"
        }
    }
}

Respond:

{
    "data": {
        "pods": []
    }
}

What you expected to happen

Instance table shows instances

How to reproduce

Only happends in one of our environments.
Not reproduced in Test environment. Maybe gone after restart.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

You could check intance_traffic table/index to last_ping, which should be updated accordingly.
We don't know any existing bugs.

[mattwmj]

We are using Elasticsearch as storage.
I don't find any index with name instance_traffic or last_ping, neither by kibana or by ES search API.
Please advice.

[wu-sheng]

https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-storage/#elasticsearch

They are in the sw_metrics-all-${day-format}.

[mattwmj]

I have created sw_metrics-all* index pattern searched records that contains intance_traffic or last_ping fields.
In a working env, there are a lot of records.
In the env with issue, there is no records at all.

Both env are configed with below TTL:

  recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:7} # Unit is day
  metricsDataTTL: ${SW_CORE_METRICS_DATA_TTL:14} # Unit is day

FYI, in the env with issue, currently we have below index in ES:

sw_metrics-all | Alias
sw_metrics-all-20230218 | Index
sw_metrics-all-20230225 | Index
sw_metrics-all-20230304 | Index

I believe that earlier indexes have been deleted by TTL.
But it is still strange that there is no intance_traffic or last_ping in existing indexes since the instance metrics works fine last month.

[wu-sheng]

This is your env, we can't tell. Sorry.

[mattwmj]

What I don't understand is that why there is no intance_traffic or last_ping in existing indexes.
Does oap server create there records when receives information from agent? Why the oap server is not creating now?

[mattwmj]

The UI is working and showing instances today without we doing anything...
The metrics shows instance data back to 02-24, so assume the data are there.

But by searching the es metics-all index I find that there are only 4 documents with metric_table: instance_traffic, whose last_ping field are all within today.

FYI, yesterday I watch the access.log and can see agents sending JVMMetrics. I assume that for some reason the oap-server is not writing instance_traffic for a period of time but is writing correctly today. I wonder what the root cause is.

[BritneyStars]

I also meet the question graphql queryInstances response empty pods. but In fact, it should be exists

[mattwmj]

This morning we find that one service is missing in service table and shows later. The latest two metric_table instance_triffic last ping records of this service are 202,305,041,114 and 202,304,270,911.

[wu-sheng]

I don't think there should be two records for one instance.

[mattwmj]

This issue happened on 8/8 again. We did some study on the log. Below are some foundings.

In oap-server log:

• 2023-08-08 00:01:18,581 DataTTLKeeperTimer removed index metrics-all-20230629 from ES as the TTL is set to 40 days.
• 2023-08-08 00:01:21,577 AliasClient reported error on get indices by alias [metrics-all] as ES API responded with no such index [metrics-all-20230629]. AliasClient kept reporting this util 2023-08-08 14:56:18,884

ES log shows that metrics-all-20230629 index was being deleted by oap and recreated by template:

• 2023-08-08T00:01:18,805][INFO ][o.e.c.m.MetadataDeleteIndexService] [elasticsearch09] [metrics-all-20230629/SYs-n9YnTMehleJ0-Oysqg] deleting index
• [2023-08-08T00:01:21,474][INFO ][o.e.c.m.MetadataCreateIndexService] [elasticsearch09] [metrics-all-20230629] creating index, cause [auto(bulk api)], templates [metrics-all], shards [2]/[1]

These delete and recreate loop kept like every five minutes util oap-server was restarted. The timestamps are:
2023-08-08T00:01:18,805 deleting
2023-08-08T00:01:21,474 creating

2023-08-08T00:06:18,610 deleting
2023-08-08T00:06:25,540 creating

2023-08-08T00:11:18,610 deleting
2023-08-08T00:11:29,344 creating

2023-08-08T00:16:18,609 deleting
2023-08-08T00:16:32,863 creating

I suspect that there is some mis match configuration on oap and ES template, but every thing on oap is default configurating by mannual and the TTL is increased to 40. Can you please share some ideas, thanks! @wu-sheng