Use config hoodie.index.scope=global|partition to avoid having variants of the same index type

[xushiyan]

Hudi supports many indexes for both read and write: simple, bloom, bucket, record, etc. And there are global variants such as global_simple, global_bloom, and partitioned variant like partitioned record index.

To simplify the usage, we can add a new config hoodie.index.scope=global|partition to eliminate the need of keeping global or partitioned variants. A legacy config like hoodie.index.type=global_simple will be equivalent to

hoodie.index.type=simple
hoodie.index.scope=global

The default index scope config can be inferred based on the index type to stay compatible with the current default behaviors of each index type.

Setting a scope config value should be validated for the index type, for e.g., bucket index is currently working on partition level, so explicitly setting the scope to global will fail the config validation.

Although it uses one more config, it makes the index names cleaner and aligned. For e.g., we have global_simple and global_bloom for global variants, but record_index is a global index with partitioned_record_index working on partition-level. The new index scope config also makes it more explicit to users about how the chosen index type behaves, and how the uniqueness should be enforced--either across partitions or within a partition.

[vinothchandar]

Interesting proposal. But the upgrade-downgrade aspects need to handled for existing usage and since its a user-facing config, code needs to be rewritten?

In effect, I feel we cannot easily make this change without continuing to support backwards compatibility. for old index types.

@jonvex @nsivabalan - what did we end up deciding for partition level RLI..

[nsivabalan]

yes. existing tables has to work w/o needing any additional writer config overrides from user. If not, it may wipe out the data or result in data consistency issues.

Vinoth:
for record index, we managed to retain the same type for hoodie.index.type which is RECORD_INDEX.
Based on the write configs set (hoodie.metadata.record.index.enable or hoodie.metadata.partitioned.record.index.enable) the indexing will figure out whether its global RLI or partitioned RLI.
but from user standpoint, they need to set hoodie.index.type to RECORD_INDEX if they prefer to use record index from MDT. And we can have only one of global rli or partitioned RLI for a given data table enabled on write.