cdc cli: support TOML output and migration-optimized diff for changefeed query

[JesseZhuang]

Is your feature request related to a problem?

cdc cli changefeed create --config consumes TOML, but cdc cli changefeed query only returns JSON. That asymmetry makes changefeed migration, config review, and version control more manual than they need to be.

Today, recreating a changefeed on another cluster looks like this:

query → JSON → jq '.config' → json-to-toml converter → fix snake_case→kebab-case
  → strip 50+ server defaults → re-add id/sink-uri/start-ts as CLI flags → create

With this feature the flow becomes native:

cdc cli changefeed query -c my-feed --only-diff -o toml > feed.toml
# fix redacted password, adjust sink-uri if needed
cdc cli changefeed create --config feed.toml

Describe the feature you'd like

This is not a config-model redesign. The scope is narrow: make export/import symmetric, reduce manual steps, and keep full backward compatibility.

changefeed query -o toml

Add --output json|toml to changefeed query, keeping json as the default. TOML output uses explicit toml:"kebab-case" struct tags so the exported file looks like a real config file. Time and duration fields use text encoding ('2026-04-18 09:35:00.139', '10m0s') instead of raw integers.

changefeed query --only-diff

Migration-focused export that prints only non-default fields. For a default changefeed:

id = 'my-feed'
sink-uri = 'blackhole:'
start-ts = 465700086583918602

For a changefeed with overrides, only the meaningful deltas appear:

id = 'my-feed'
sink-uri = 'mysql://root:xxxxx@downstream:4000/'
start-ts = 434266532874690561

[config]
  case-sensitive = true

[config.filter]
  rules = [
    'db1.*',
    '!db1.audit_log',
  ]

[config.scheduler]
  enable-table-across-nodes = true
  region-threshold = 1000

The diff is map-based (serialize both current and default configs through the same codec, then recursively compare), so it requires zero maintenance when new config fields are added upstream.

The exported TOML is round-trip consistent: create from the export, query --only-diff again, get identical output (given the same start-ts).

Top-level changefeed fields in the config file

The TOML file can carry id, sink-uri, start-ts, target-ts, and keyspace alongside [config], making it a self-contained artifact. CLI flags take precedence when both are provided.

cdc cli changefeed create --config feed.toml    # no extra CLI flags needed
cdc cli changefeed update --config feed.toml    # id read from the file

Backward-compatible config parsing

The parser accepts both legacy top-level config style and canonical [config] style, but rejects mixing them in the same file. Strict decoding catches typos like case_sensitive vs case-sensitive before the changefeed is created. JSON output, legacy config files, and all existing CLI flags remain unchanged.

Describe alternatives you've considered

The current workaround uses an external pipeline:

cdc cli changefeed query -c feed | jq '.config' > config.json
./json2toml config.json > config.toml
# then manually: fix snake_case→kebab-case, strip defaults, re-add id/sink-uri/start-ts

This requires external tooling (jq, custom converter binary), does not strip server-injected defaults, loses top-level fields like id and sink-uri, does not validate field names, and does not guarantee round-trip consistency.

https://github.com/pingcap/ticdc/tree/master/cmd/config-converter

Natively solving this inside the CLI eliminates the tooling dependency and makes the export artifact usable as-is.

Teachability, Documentation, Adoption, Migration Strategy

Adoption and migration

• JSON output remains the default — zero behavior change for existing users and scripts.
• Legacy flat-style TOML config files continue to work.
• All existing CLI flags and subcommands are unchanged.
• The contribution is purely additive; no breaking changes.

Implementation

I have a working implementation (~2,500 lines across 20 files including tests) validated on both classic TiCDC v8.1.2 and new-arch TiCDC v8.5.6. Happy to submit as one PR or a series:

1. TOML tags + time/duration text marshaling (pure additive, zero behavior change)
2. --output toml on query
3. --only-diff with map-based diff
4. Config file enhancements (top-level fields, mixed-style rejection, update improvements)

Integration tests

The implementation includes 16+ end-to-end integration tests run against live TiCDC clusters, covering:

• Default and override-heavy changefeed export (JSON + TOML)
• TOML-only create (no CLI args) and CLI-overrides-TOML precedence
• Mixed config style rejection (4 rejection cases + acceptance cases)
• Full TOML round-trip: export → edit → import → verify identical output
• Large filter array preservation and exact diff verification
• Update path: id/sink-uri/keyspace from TOML, merge semantics, error cases

A detailed test matrix and the integration test artifact are attached.

integration.tests.20260420.zip

Questions

I would appreciate feedback before preparing the upstream PR:

1. Does this direction look reasonable for TiCDC CLI?
2. Is --only-diff the right flag name, or would --non-default / --minimal be better?
3. Single PR or PR series for ~2,500 lines?
4. For changefeed update --config, should omitted fields preserve existing values (merge) or reset to defaults (replace)?

Context

This was developed at Pinterest to solve operational pain points we encounter as a TiDB open-source customer running TiCDC for changefeed migration, environment cutover, and configuration review. Happy to iterate based on feedback.

[ti-chi-bot]

Welcome @JesseZhuang! It looks like this is your first issue to pingcap/ticdc 🎉