Skip to content

[Improve]:(MongoDB)optimize type inference to avoid unnecessary decimal conversion#596

Merged
JNSimba merged 1 commit into
apache:masterfrom
kwonder0926:dev
May 12, 2025
Merged

[Improve]:(MongoDB)optimize type inference to avoid unnecessary decimal conversion#596
JNSimba merged 1 commit into
apache:masterfrom
kwonder0926:dev

Conversation

@kwonder0926
Copy link
Copy Markdown
Contributor

Proposed changes

Previously, all numeric types were converted to Decimal to prevent precision loss when syncing mixed-type fields from MongoDB to Doris. This caused unnecessary precision degradation for single-type fields. For example, if the initial table has a small amount of data and an INT-type field with all values below 1000, after conversion to DECIMAL, the numeric precision will be limited to 3 digits. If the data evolves with business growth and the field value later exceeds 1000 (e.g., becomes a 4-digit number), it will lose precision due to the previously inferred limited length.

Problem Summary:

This commit improves the type inference logic:

  • Only convert to Decimal if multiple types are present in the sampled data;
  • Retain original types (e.g., INT, BIGINT) for fields with consistent data types;

Checklist(Required)

  1. Does it affect the original behavior: (No)
  2. Has unit tests been added: (Yes)
  3. Has document been added or modified: (No Need)
  4. Does it need to update dependencies: (No)
  5. Are there any changes that cannot be rolled back: (Yes)

Further comments

Copy link
Copy Markdown
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit 8f0052c into apache:master May 12, 2025
6 checks passed
@JNSimba JNSimba mentioned this pull request Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants