Skip to content

feat: add iceberg storage consumer support - #105

Closed
tenfyzhong wants to merge 17 commits into
pingcap-inc:mainfrom
tenfyzhong:docs/iceberg-storage-consumer-design
Closed

feat: add iceberg storage consumer support#105
tenfyzhong wants to merge 17 commits into
pingcap-inc:mainfrom
tenfyzhong:docs/iceberg-storage-consumer-design

Conversation

@tenfyzhong

Copy link
Copy Markdown
Collaborator

Summary

  • add a parallel Iceberg consumer path with metadata-version replay, row conversion, checkpoints, and shared row staging while keeping the existing CSV path intact
  • wire --source-format=iceberg and --iceberg.source-uri into the warehouse commands for Snowflake, Redshift, BigQuery, and Databricks
  • add warehouse-specific row appliers, Iceberg metrics, docs, and replace the local TiCDC worktree override with a reproducible fork reference

Changes

  • introduce pkg/icebergconsumer for config, checkpoints, schema diffing, and row conversion
  • add ReplicateIceberg(...) runtime wiring in cmd/core.go and source flag validation for Iceberg mode
  • add row appliers for Snowflake, Redshift, BigQuery, and Databricks
  • migrate compatible imports from legacy TiFlow paths to TiCDC paths
  • update docs for Iceberg source usage and staging semantics

Testing

  • GOTOOLCHAIN=go1.25.8 go test ./... -count=1 in an environment that rewrites https://github.com/ to SSH and sets GOPROXY=direct, GOPRIVATE=github.com/tenfyzhong/ticdc, GONOSUMDB=github.com/tenfyzhong/ticdc, and GONOPROXY=github.com/tenfyzhong/ticdc so the forked TiCDC replacement can be fetched reproducibly

Dependency

Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
@tenfyzhong tenfyzhong closed this May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant