Dirty data filtering function脏数据过滤功能

[tusiCHN]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

脏数据过滤功能。比如一共有100条数据处理，只有一条脏数据，脏数据比例为1%，或者1小时100条数据，也只有一条脏数据。这种情况可以先保证正常数据进去，脏数据抛出异常，或者告警。

希望可以通过总数据量或者时间范围两个维度去设置允许脏数据比例，或者总脏数据数量。

Dirty data filtering function. For example, there are 100 data processing, only one dirty data, the proportion of dirty data is 1%, or 100 data an hour, only one dirty data. In this case, you can ensure that normal data is imported, and dirty data is generated abnormally, or an alarm is generated.

It is hoped that the proportion of allowed dirty data or the total number of dirty data can be set by the total data amount or time range.

Usage Scenario

在为实时业务提供数据支持时，因为一条脏数据，导致任务失败。且长时间未发现任务已经停止。

When providing data support for real-time services, a dirty piece of data caused a task failure. Procedure The task has not been stopped for a long time.

Related issues

是的

yes

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dik111]

+1

[peigenxiao]

+2 因为源端和目标端数据类型不一致，脏数据无法彻底避免。可以参考kettle把脏数据进行记录，同时过滤或者替换脏数据，确保同步任务顺利完成。Because the data types of the source and target are inconsistent, dirty data cannot be completely avoided. You can refer to Kettle to record dirty data and filter or replace it to ensure the smooth completion of synchronization tasks.

[davidzollo]

@tusiCHN Would you like to design this feature? #4587

[Xuxiaotuan]

@tusiCHN Would you like to design this feature? #4587

I want to try to implement this function

[KangTomwk]

@tusiCHN Would you like to design this feature? #4587

I want to try to implement this function

Hello, is there any substantial progress on this?