metrics: show Disconnection Count as a rate, split out undetermined

[dulao5]

What problem does this PR solve?

Issue Number: close #70336

Problem Summary:

The Server row Disconnection Count panel in the TiDB Grafana dashboard plotted the raw cumulative tidb_server_disconnection_total counter. On a long-running production cluster this counter accumulates to very large numbers, so it stops being useful for spotting current connect/disconnect activity.

What changed and how does it work?

• Switched the ok/error result series to irate(...[30s]), so the panel reflects disconnect frequency (and, transitively, connection churn / whether the workload looks long- or short-lived).
• Kept result="undetermined" as its own series, still a cumulative total, routed to the panel's right Y axis via a seriesOverrides entry (unstacked). Any nonzero undetermined count means a transaction was once left in a state TiDB could not automatically resolve and needs manual investigation, so it stays an absolute counter instead of being folded into a rate.
• Applied the same change to the two next-gen counterparts of this panel (pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json, pkg/metrics/nextgengrafana/tidb_worker.json) to keep all three dashboard definitions consistent.

Manually verified by loading all three dashboard JSON files into a local tiup playground Grafana instance and confirming the panel renders the rate view plus the separate right-axis undetermined series as expected.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manually loaded the three modified dashboard JSON files into a local tiup playground Grafana instance (http://127.0.0.1:3000) and confirmed the Disconnection Count panel renders correctly.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Enhancements
  • Updated TiDB disconnection Grafana panels to show known-result disconnections as 30-second rates.
  • Displayed undetermined disconnections separately as cumulative counts on a secondary axis without stacking.
  • Improved panel descriptions and visualization settings to distinguish rates, cumulative totals, and undetermined results.
  • Applied consistent metric presentation across standard, keyspace-aware, and worker dashboards.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yibin87 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/metrics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

Hi @dulao5. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

Welcome @dulao5!

It looks like this is your first PR to pingcap/tidb 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/tidb. 😃

[pingcap-cla-assistant]

All committers have signed the CLA.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bf56ee55-509c-4e78-b713-8c7cdf8a2e73

📥 Commits

Reviewing files that changed from the base of the PR and between 66045cd and e12fb90.

📒 Files selected for processing (3)

• pkg/metrics/grafana/tidb.json
• pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json
• pkg/metrics/nextgengrafana/tidb_worker.json

🚧 Files skipped from review as they are similar to previous changes (3)

• pkg/metrics/grafana/tidb.json
• pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json
• pkg/metrics/nextgengrafana/tidb_worker.json

---

📝 Walkthrough

Walkthrough

The three TiDB Grafana panels now display known disconnection results as 30-second irate values. They display undetermined results as separate cumulative totals on the right axis.

Changes

TiDB disconnection panel updates

Layer / File(s)	Summary
Panel queries and axis presentation pkg/metrics/grafana/tidb.json, pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json, pkg/metrics/nextgengrafana/tidb_worker.json	Panel descriptions document rate-based known results and cumulative undetermined totals. Queries use irate(...[30s]) for known results and route cumulative undetermined series to a non-stacked secondary axis.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit split the counter streams,
Known results race in measured beams.
Undetermined totals stay in view,
On the right axis, clear and true.
Grafana records each hop.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main dashboard change: showing Disconnection Count as a rate and separating undetermined results.
Description check	✅ Passed	The description includes the issue, problem, implementation details, testing information, side effects, documentation status, and release note.
Linked Issues check	✅ Passed	The changes satisfy issue #70336 by adding 30-second rates, preserving cumulative totals, separating undetermined results, and updating all three dashboards.
Out of Scope Changes check	✅ Passed	All changes are limited to the three Disconnection Count dashboard definitions required by issue #70336.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

pkg/metrics/grafana/tidb.json (1)

3137-3148: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the now-dead total series override.

The {"alias": "total", "fill": 0, "lines": false} override no longer matches any series legend. Target A now emits {{instance}}-{{result}} and target B now emits {{instance}}-undetermined-total. Neither legend equals the literal string total. Remove the override to avoid confusing future maintainers who might assume a total series still exists.

♻️ Proposed cleanup

           "seriesOverrides": [
-            {
-              "alias": "total",
-              "fill": 0,
-              "lines": false
-            },
             {
               "alias": "/-undetermined-total$/",
               "stack": false,
               "yaxis": 2
             }
           ],

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/metrics/grafana/tidb.json` around lines 3137 - 3148, Remove the dead
series override with alias "total" from the seriesOverrides configuration, while
preserving the "/-undetermined-total$/" override unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/metrics/grafana/tidb.json`:
- Around line 3137-3148: Remove the dead series override with alias "total" from
the seriesOverrides configuration, while preserving the "/-undetermined-total$/"
override unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 055e42f8-65ba-4ccb-96d0-f053491d7137

📥 Commits

Reviewing files that changed from the base of the PR and between 8380b57 and 56c6c2a.

📒 Files selected for processing (3)

• pkg/metrics/grafana/tidb.json
• pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json
• pkg/metrics/nextgengrafana/tidb_worker.json

[dulao5]

/ok-to-test

[ti-chi-bot]

@dulao5: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yibin87]

/ok-to-test

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.3791%. Comparing base (6cbbd22) to head (e12fb90).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #70337        +/-   ##
================================================
- Coverage   76.3271%   73.3791%   -2.9480%     
================================================
  Files          2041       2079        +38     
  Lines        559032     584768     +25736     
================================================
+ Hits         426693     429098      +2405     
- Misses       131439     155150     +23711     
+ Partials        900        520       -380     

Flag	Coverage Δ
integration	40.6853% <ø> (+1.0166%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	59.9240% <ø> (ø)
parser	∅ <ø> (∅)
br	46.6167% <ø> (-16.0923%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dulao5]

/retest-required

[dulao5]

/retest-required

[dulao5]

/retest

[yibin87]

Please preserve the existing target and add the new rate query separately. Replacing it would change the existing Disconnection Count semantics and drop the cumulative count view.

[dulao5]

Updated per review feedback: restored the original sum(tidb_server_disconnection_total{...}) by (instance, result) target unchanged (refId A) so the existing cumulative view and dashboard semantics are preserved.

The new views are now additive, on the right Y axis:

• refId B: sum(irate(tidb_server_disconnection_total{...,result!="undetermined"}[30s])) by (instance, result) — ok/error disconnect rate
• refId C: sum(tidb_server_disconnection_total{...,result="undetermined"}) by (instance) — undetermined cumulative counter, kept separate since any nonzero value needs investigation

Applied consistently across all three affected dashboard files.

[dulao5]

Dropped the redundant result="undetermined" query — it duplicated the undetermined series already produced by the original by (instance, result) cumulative target. Now that series is just routed to the right Y axis via a seriesOverrides alias match (/-undetermined$/), no extra target needed.

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-tests-checked label, please finished the tests then check the finished items in description.

For example:

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code

‼️ Must keep the HTML comments <!-- At least one of them must be included. -->

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}