ddl: Filter the DDL Job History based on DDL types #54389

sachin-japate · 2024-07-02T09:45:16Z

Adding the capability to filter the DDL Job History based on DDL types

What problem does this PR solve?

This PR extends the DDL history job API capability to filter the DDL jobs based on their types.

Issue Number: close #54391

Problem Summary:

Requirements raised due to issues with incremental backups in TiDB have some limitations, incremental backup after certain types of DDL executed can lead to data inconsistency issues.
To solve this, we wanted to handle this situation with our backup scheduling platform, We will introduce a validation before triggering any incremental backup if we encounter such DDLs, full backup will be triggered instead of incremental.

Currently, ddl/history API can return all DDLs or DDLs with a starting job_id.
http://{TiDBIP}:10080/ddl/history
http://{TiDBIP}:10080/ddl/history?limit={number}
http://{TIDBIP}:10080/ddl/history?start_job_id={id}&limit={number}

Extending API to filter based on the DDL types.
http://{TIDBIP}:10080/ddl/history?ddl_types="2,4,47"
http://{TIDBIP}:10080/ddl/history?ddl_types="2,4,47"&limit={number}
http://{TIDBIP}:10080/ddl/history?start_job_id={id}&ddl_types="2,4,47"&limit={number}

What changed and how does it work?

Filter the DDL job history based on the DDL types,

http://{TIDBIP}:10080/ddl/history?ddl_types="2,4,47"

Filter the DDL job history based on the DDL types and apply the limit

http://{TIDBIP}:10080/ddl/history?ddl_types="2,4,47"&limit={number}

Filter the DDL job history based on the DDL types from the starting job ID and apply the limit

http://{TIDBIP}:10080/ddl/history?start_job_id={id}&ddl_types="2,4,47"&limit={number}

the combination of start_job_id and ddl_types will not work without limit, as the previous implementation was if the start_job_id was sent limit was a mandatory param. So behavior will remain the same as previously.

Check List

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test

Documentation

Release note

None

This reverts commit c0f4c5d.

ti-chi-bot · 2024-07-02T09:45:18Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sre-bot · 2024-07-02T09:45:23Z

All committers have signed the CLA.

ti-chi-bot · 2024-07-02T09:45:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tangenta for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-07-02T09:45:25Z

Welcome @sachin-japate!

It looks like this is your first PR to pingcap/tidb 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to pingcap/tidb. 😃

ti-chi-bot · 2024-07-02T09:45:25Z

Hi @sachin-japate. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2024-07-02T09:45:34Z

Hi @sachin-japate. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wuhuizuo · 2024-07-02T09:55:05Z

/ok-to-test

sachin-japate · 2024-07-02T10:13:58Z

/retest

codecov · 2024-07-02T10:17:33Z

Codecov Report

Attention: Patch coverage is 0% with 51 lines in your changes missing coverage. Please review.

Project coverage is 55.8435%. Comparing base (ebc9620) to head (ccba9d5).
Report is 20 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #54389         +/-   ##
=================================================
- Coverage   72.8286%   55.8435%   -16.9852%     
=================================================
  Files          1530       1648        +118     
  Lines        435981     609940     +173959     
=================================================
+ Hits         317519     340612      +23093     
- Misses        98821     246706     +147885     
- Partials      19641      22622       +2981

Flag	Coverage Δ
integration	`35.3724% <0.0000%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9656% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`46.2404% <ø> (+0.0987%)`	⬆️

sachin-japate · 2024-07-02T10:41:04Z

/test unit-test

tiprow · 2024-07-02T10:41:26Z

@sachin-japate: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lance6716

Thansk for your contribution. Can you add some tests to better show the usage? You can refer to

tidb/pkg/server/handler/tests/http_handler_test.go

Line 971 in 9960ae7

func TestAllHistory(t *testing.T) {

D3Hunter · 2024-07-02T11:47:25Z

/hold

features require discussion first

ti-chi-bot · 2024-07-03T11:09:30Z

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-tests-checked label, please finished the tests then check the finished items in description.

For example:

Tests

Unit test

Integration test

Manual test (add detailed scripts or steps below)

No code

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}

tiprow · 2024-07-03T11:27:13Z

@sachin-japate: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`ccba9d5`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2024-07-03T11:48:16Z

@sachin-japate: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/check_dev_2	`ccba9d5`	link	true	`/test check-dev2`
idc-jenkins-ci-tidb/unit-test	`ccba9d5`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sachin-japate added 4 commits July 1, 2024 23:18

Extending ddl history api to fitlter jobs based on type

c0f4c5d

Revert "Extending ddl history api to fitlter jobs based on type"

f019ad6

This reverts commit c0f4c5d.

Extending ddl history api to fitlter jobs based on type

565ada5

commit fix

c8c303b

ti-chi-bot bot added do-not-merge/invalid-title do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 2, 2024

ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 2, 2024

ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jul 2, 2024

sachin-japate changed the title ~~Adding the capability to filter the DDL Job History based on DDL types~~ Filter the DDL Job History based on DDL types Jul 2, 2024

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Jul 2, 2024

sachin-japate changed the title ~~Filter the DDL Job History based on DDL types~~ ddl: Filter the DDL Job History based on DDL types Jul 2, 2024

ti-chi-bot bot removed the do-not-merge/invalid-title label Jul 2, 2024

lance6716 reviewed Jul 2, 2024

View reviewed changes

revert change

8693fd0

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 2, 2024

sachin-japate added 3 commits July 3, 2024 13:12

fix tests

9dd0563

fix tests

82d9f9f

removed assignment to the blank identifier

ccba9d5

sachin-japate closed this Jul 3, 2024

sachin-japate deleted the filter-ddlhistory branch July 3, 2024 12:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: Filter the DDL Job History based on DDL types #54389

ddl: Filter the DDL Job History based on DDL types #54389

sachin-japate commented Jul 2, 2024 •

edited by D3Hunter

Loading

ti-chi-bot bot commented Jul 2, 2024

sre-bot commented Jul 2, 2024 •

edited

Loading

ti-chi-bot bot commented Jul 2, 2024

ti-chi-bot bot commented Jul 2, 2024

ti-chi-bot bot commented Jul 2, 2024

tiprow bot commented Jul 2, 2024

wuhuizuo commented Jul 2, 2024

sachin-japate commented Jul 2, 2024

codecov bot commented Jul 2, 2024 •

edited

Loading

sachin-japate commented Jul 2, 2024

tiprow bot commented Jul 2, 2024

lance6716 left a comment

D3Hunter commented Jul 2, 2024

ti-chi-bot bot commented Jul 3, 2024

tiprow bot commented Jul 3, 2024

ti-chi-bot bot commented Jul 3, 2024 •

edited

Loading

ddl: Filter the DDL Job History based on DDL types #54389

ddl: Filter the DDL Job History based on DDL types #54389

Conversation

sachin-japate commented Jul 2, 2024 • edited by D3Hunter Loading

Adding the capability to filter the DDL Job History based on DDL types

What problem does this PR solve?

Problem Summary:

What changed and how does it work?

Check List

Documentation

Release note

ti-chi-bot bot commented Jul 2, 2024

sre-bot commented Jul 2, 2024 • edited Loading

ti-chi-bot bot commented Jul 2, 2024

ti-chi-bot bot commented Jul 2, 2024

ti-chi-bot bot commented Jul 2, 2024

tiprow bot commented Jul 2, 2024

wuhuizuo commented Jul 2, 2024

sachin-japate commented Jul 2, 2024

codecov bot commented Jul 2, 2024 • edited Loading

Codecov Report

sachin-japate commented Jul 2, 2024

tiprow bot commented Jul 2, 2024

lance6716 left a comment

Choose a reason for hiding this comment

D3Hunter commented Jul 2, 2024

ti-chi-bot bot commented Jul 3, 2024

tiprow bot commented Jul 3, 2024

ti-chi-bot bot commented Jul 3, 2024 • edited Loading

sachin-japate commented Jul 2, 2024 •

edited by D3Hunter

Loading

sre-bot commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 2, 2024 •

edited

Loading

ti-chi-bot bot commented Jul 3, 2024 •

edited

Loading