Introduce new fields 'delete_table' and 'delete_partition' in UpdateMetadata RPC

[swuferhong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

In pr #890, we will use TableMetadata#TABLE_DURATION_DELETE_PATH or TableMetadata#TABLE_DURATION_DELETE_ID to represent a table that needs to be deleted, and use PartitionMetadata#PARTITION_DURATION_DELETE_NAME or PartitionMetadata#PARTITION_DURATION_DELETE_ID a partition that needs to be deleted. Although this approach works, but it is difficult to understand. New fields 'delete_table' and 'delete_partition' can be introduced to represent it, but backward compatibility needs to be considered

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[caozhen1937]

@swuferhong I want to try it after #890 merged, can you assign it to me?

[swuferhong]

@swuferhong I want to try it after #890 merged, can you assign it to me?

Hi, @caozhen1937. We can continue using names like TableMetadata #TABLE_DURATION_DELETE_PATH, but we need to prohibit these special table and partition names during creation. For example: We could design:
TABLE_DURATION_DELETE_PATH as ___internal___deleted_table___ (Three underline lines)
PARTITION_DURATION_DELETE_NAME as ___internal___deleted_partition___ (Three underline lines)

During table/partition creation validation, we should check if the name starts with ___internal___. If it does, Fluss should throw an error. Can you do that work after #890 merged. Thanks.

[caozhen1937]

@swuferhong Get, I will validate the design, and work after #890 merged.

[swuferhong]

@swuferhong Get, I will validate the design, and work after #890 merged.

Hi, @caozhen1937 as #890 has been merged, do u want take this issue?

[caozhen1937]

@swuferhong Get, I will validate the design, and work after #890 merged.

Hi, @caozhen1937 as #890 has been merged, do u want take this issue?

@swuferhong Yes, please assign it to me.

[wuchong]

We can simplify the design to just use the prefix __ for the internal purpose, which keeps alignment and is compatible with Kafka (e.g., __consumer_offsets in the future), and forbids any table/partition creation that begins with this prefix.

So we can use __deleted_marker__ as the special marker for deletion.

[caozhen1937]

After discussing offline with @swuferhong , I will create another issue will verify that table names or partition names cannot start with __.

Regarding the issue of explicitly marked deletion of table or partition, we will check if anyone is working on PbTableMetadataV2 to avoid conflicts.