cdc changefeed failed with "[CDC:ErrS3StorageAPI]external storage api: operation error S3: DeleteObjects, https response error StatusCode: 400, RequestID: xxx, HostID: xxx, api error MissingContentMD5: Missing required header for this request"

[Lily2025]

What did you do?

1，create changefeed BDR with redo config
config: "bdr-mode = true\nmemory-quota = 2147483648\n[consistent]\nlevel = "eventual"\nflush-interval = 500\nstorage = "s3://tmp/test-changefeed1?access-key=xxx&secret-access-key=xxx&endpoint=http%3a%2f%2fminio-peer%3a9000""
2，kill pd leader

What did you expect to see?

changefeed task is normal

What did you see instead?

changefeed task failed with error

./cdc cli changefeed query --server=http://127.0.0.1:8301 --changefeed-id=ticdc-task1
{
"upstream_id": 7651353003413274603,
"id": "ticdc-task1",
"keyspace": "",
"sink_uri": "mysql://root:xxxxx@cdc-downstream-tidb.endless-ha-test-bdr-tps-8146357-1-5:4000",
"config": {
"memory_quota": 2147483648,
"case_sensitive": false,
"force_replicate": false,
"ignore_ineligible_table": true,
"check_gc_safe_point": true,
"enable_sync_point": false,
"enable_table_monitor": false,
"bdr_mode": true,
"sync_point_interval": 600000000000,
"sync_point_retention": 86400000000000,
"filter": {
"rules": [
"."
]
},
"mounter": {
"worker_num": 16
},
"sink": {
"delete_only_output_handle_key_columns": null,
"content_compatible": null,
"advance_timeout": 150,
"send_bootstrap_interval_in_sec": 120,
"send_bootstrap_in_msg_count": 10000,
"send_bootstrap_to_all_partition": true,
"send-all-bootstrap-at-start": false,
"debezium_disable_schema": false,
"debezium": {
"output_old_value": true
},
"open": {
"output_old_value": true
}
},
"consistent": {
"level": "eventual",
"max_log_size": 64,
"flush_interval": 500,
"meta_flush_interval": 200,
"encoding_worker_num": 16,
"flush_worker_num": 8,
"storage": "s3://tmp/test-changefeed1?access-key=minioadmin\u0026secret-access-key=minioadmin\u0026endpoint=http%3a%2f%2fminio-peer%3a9000",
"use_file_backend": false,
"compression": "none",
"memory_usage": {
"memory_quota_percentage": 50
}
},
"scheduler": {
"enable_table_across_nodes": false,
"region_threshold": 10000,
"write_key_threshold": 0
},
"integrity": {
"integrity_check_level": "none",
"corruption_handle_level": "warn"
},
"changefeed_error_stuck_duration": 1800000000000,
"synced_status": {
"synced_check_interval": 300,
"checkpoint_interval": 15
}
},
"create_time": "2026-06-15 05:12:48.278",
"start_ts": 467002082770026502,
"resolved_ts": 0,
"target_ts": 0,
"checkpoint_tso": 467002320586801157,
"checkpoint_time": "2026-06-15 05:27:55.383",
"state": "failed",
"error": {
"time": "0001-01-01T00:00:00Z",
"addr": "tc-ticdc-0.tc-ticdc-peer.endless-ha-test-bdr-tps-8146357-1-5.svc.cluster.local:8301",
"code": "CDC:ErrS3StorageAPI",
"message": "[CDC:ErrS3StorageAPI]external storage api: operation error S3: DeleteObjects, https response error StatusCode: 400, RequestID: 18B91191D6334B5E, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error MissingContentMD5: Missing required header for this request: Content-Md5."
},
"creator_version": "v8.5.7"
}
sh-5.1#

Versions of the cluster

./cdc version
Release Version: v8.5.7
Git Commit Hash: bdef55758cf67f669c59db99484779cdf967a171
Git Branch: HEAD
UTC Build Time: 2026-06-12 07:52:01
Go Version: go version go1.25.10 linux/amd64
Failpoint Build: false
Kernel Type: Classic
2026-06-15T05:53:09.788+0800

[wk989898]

The storage should have an explicit provider parameter.
see pingcap/tidb#65326

[Lily2025]

/reopen

[ti-chi-bot]

@Lily2025: Reopened this issue.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[lilinghai]

When using ks3, you also need to add the provider (s3:// + provider=ks3) parameter to bypass this problem.