Skip to content

sync-diff-inspector silently swallows TiDB row stream errors and generates false REPLACE fix SQL #900

Description

@banerjeeadrish

Bug Report

  1. What did you do?

Ran sync diff inspector against two copies of a database, one on MySQL and one on TiDB. The database had a table with JSON columns. Because MySQL and TiDB can serialize JSON keys in different orders, the chunk checksum reported a mismatch and sync-diff-inspector entered compareRows(...) to compare rows directly.

During the TiDB-side row fetch, the TiDB query hit an error mid-stream. However, TiDBRowsIterator.Next() currently treats rows.Next() == false as clean EOF and does not check rows.Err(). As a result, the TiDB query error is silently swallowed. The comparison code then assumes the downstream TiDB iterator is exhausted and reports the remaining upstream MySQL rows as missing on TiDB, producing false diffs and bogus fix SQL. Ideally, the tool should surface the original TiDB query error and fail the table comparison instead.

Reproduction

One deterministic way to reproduce is to force the TiDB query to exceed a very small memory quota.
Pick a table that sync-diff-inspector will row-compare. A table with a JSON column is useful because JSON serialization differences can force checksum mismatch and make compareRows() run.

Configure a tiny per-query TiDB memory quota:

[data-sources.tidb.session]
tidb_mem_quota_query = 1048576
tidb_mem_oom_action = "CANCEL"

Run:

sync_diff_inspector -C config.toml

The TiDB query will be cancelled with an error like:

Error 8175 (HY000): Your query has been cancelled due to exceeding the allowed memory
limit for a single SQL query ... increase the tidb_mem_quota_query limit

But sync-diff-inspector does not surface the error. Instead, it can report +N/-0 differences and write REPLACE INTO ... SQL for rows that already exist on both sides.

Root cause is the current TiDB iterator Next() impl doesn't handle errors

  1. What did you expect to see?

sync-diff-inspector should surface the TiDB row stream error and mark the table comparison as failed.
It should not treat the failed TiDB query as EOF, should not report downstream rows as missing, and should not generate fix SQL for that table.

  1. What did you see instead?

sync-diff-inspector silently treats the TiDB row-stream error as clean EOF.

The result will be:

  1. Table reported as not equal with +N/-0

  2. identical upstream/downstream COUNT(*) in the summary

  3. no TiDB query error surfaced

  4. no per-column diff logged

  5. fix-on-tidb/*.sql contains bogus REPLACE INTO ... statements for rows that already exist on both sides

  6. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

8.5.2

  1. which tool are you using?

sync-diff-inspector

  1. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?
./bin/sync_diff_inspector -V
App Name: sync_diff_inspector
Release Version: v8.5.5-2-g1b99c8c-dirty
Git Commit Hash: 1b99c8cd02113763ef5659c0975c9b1aefbd7faa

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions