Unstable Test TableChangeWatcherTest

[loserwang1024]

Search before asking

• I searched in the issues and found nothing similar.

Description

TableChangeWatcherTest sometime will run in failed. In log, it seems that the later test will be influenced by previous test.

Take for example:TableChangeWatcherTest.testSchemaChanges will receive the node modifaction of TableChangeWatcherTest.testPartitionedTable because CreatePartitionEvent{tablePath=db.partition_table, tableId=10, partitionName='2011', partitionId=1, partitionAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}}

Error:    TableChangeWatcherTest.testSchemaChanges:282->lambda$testSchemaChanges$4:286 
Expecting actual:
  [CreatePartitionEvent{tablePath=db.partition_table, tableId=10, partitionName='2011', partitionId=1, partitionAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    CreatePartitionEvent{tablePath=db.partition_table, tableId=10, partitionName='2022', partitionId=2, partitionAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.partition_table, schemaInfo=SchemaInfo{id=1, schema=(a INT,b STRING)}},
    SchemaChangeEvent{tablePath=db.table_0, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_0, tableId=11, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064493, modifiedTime=1766567064493}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.table_1, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_1, tableId=12, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064501, modifiedTime=1766567064501}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.table_2, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_2, tableId=13, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064507, modifiedTime=1766567064507}, tableAssignment={0=[2, 1, 0], 1=[0, 2, 1], 2=[1, 0, 2]}},
    SchemaChangeEvent{tablePath=db.table_3, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_3, tableId=14, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064514, modifiedTime=1766567064514}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_4, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_4, tableId=15, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064521, modifiedTime=1766567064521}, tableAssignment={0=[2, 0, 1], 1=[0, 1, 2], 2=[1, 2, 0]}},
    SchemaChangeEvent{tablePath=db.table_5, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_5, tableId=16, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064528, modifiedTime=1766567064528}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_6, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_6, tableId=17, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064534, modifiedTime=1766567064534}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_7, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_7, tableId=18, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064540, modifiedTime=1766567064540}, tableAssignment={0=[1, 0, 2], 1=[2, 1, 0], 2=[0, 2, 1]}},
    SchemaChangeEvent{tablePath=db.table_8, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_8, tableId=19, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064548, modifiedTime=1766567064548}, tableAssignment={0=[2, 1, 0], 1=[0, 2, 1], 2=[1, 0, 2]}},
    SchemaChangeEvent{tablePath=db.table_9, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_9, tableId=20, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064554, modifiedTime=1766567064554}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}}]
to contain exactly in any order:
  [CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_0, tableId=11, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064494, modifiedTime=1766567064494}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.table_0, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_1, tableId=12, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064502, modifiedTime=1766567064502}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.table_1, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_2, tableId=13, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064508, modifiedTime=1766567064508}, tableAssignment={0=[2, 1, 0], 1=[0, 2, 1], 2=[1, 0, 2]}},
    SchemaChangeEvent{tablePath=db.table_2, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_3, tableId=14, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064515, modifiedTime=1766567064515}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_3, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_4, tableId=15, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064522, modifiedTime=1766567064522}, tableAssignment={0=[2, 0, 1], 1=[0, 1, 2], 2=[1, 2, 0]}},
    SchemaChangeEvent{tablePath=db.table_4, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_5, tableId=16, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064529, modifiedTime=1766567064529}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_5, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_6, tableId=17, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064535, modifiedTime=1766567064535}, tableAssignment={0=[0, 2, 1], 1=[1, 0, 2], 2=[2, 1, 0]}},
    SchemaChangeEvent{tablePath=db.table_6, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_7, tableId=18, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064541, modifiedTime=1766567064541}, tableAssignment={0=[1, 0, 2], 1=[2, 1, 0], 2=[0, 2, 1]}},
    SchemaChangeEvent{tablePath=db.table_7, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_8, tableId=19, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064549, modifiedTime=1766567064549}, tableAssignment={0=[2, 1, 0], 1=[0, 2, 1], 2=[1, 0, 2]}},
    SchemaChangeEvent{tablePath=db.table_8, schemaInfo=SchemaInfo{id=1, schema=(a INT)}},
    CreateTableEvent{tableInfo=TableInfo{tablePath=db.table_9, tableId=20, schemaId=1, schema=(a INT), physicalPrimaryKeys=[], bucketKeys=[a], partitionKeys=[], numBuckets=3, properties={table.replication.factor=3}, customProperties={}, comment='null', createdTime=1766567064555, modifiedTime=1766567064555}, tableAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.table_9, schemaInfo=SchemaInfo{id=1, schema=(a INT)}}]
but the following elements were unexpected:
  [CreatePartitionEvent{tablePath=db.partition_table, tableId=10, partitionName='2011', partitionId=1, partitionAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    CreatePartitionEvent{tablePath=db.partition_table, tableId=10, partitionName='2022', partitionId=2, partitionAssignment={0=[1, 2, 0], 1=[2, 0, 1], 2=[0, 1, 2]}},
    SchemaChangeEvent{tablePath=db.partition_table, schemaInfo=SchemaInfo{id=1, schema=(a INT,b STRING)}}]

Willingness to contribute

• I'm willing to submit a PR!

[vamossagar12]

Is this related to just thetestPartitionedTable? I see that we have event watchers but at the end of each test,

    @AfterEach
    void after() {
        if (tableChangeWatcher != null) {
            tableChangeWatcher.stop();
        }
    }

we stop the tableChangeWatcher. Should we also invoke TestingEventManager.clearEvents to ensure all events are cleared? This way, we shouldn't be dependent on the event change listener to be processed, in case some events can't be processed?

[vamossagar12]

@loserwang1024 , I created this PR: #2249 PTAL.

[loserwang1024]

Hi, @vamossagar12 , each test will create a new eventManager. I don't think TestingEventManager.clearEvents can solve it. But I haven't found out why.
Maybe we can observe more fail test to analyze the reason.

I think the reason is that testPartitionedTable is also failed ,thus won't exec its metadataManager.dropTable(tablePath, false). When testSchemaChanges is started, CreatePartitionEvent is be read again. But I lack the log.

[loserwang1024]

I have reproduce the test in a simple way:

Why testPartitionedTable is failed, I remember is lack CreateTableEvent? I still try to find out the reason.