Feature: databend-meta: sequential key to make it a WAL

[drmingdrmer]

Summary

Make databend-meta a distributed WAL to support distributed transactional write in databend.

A distributed transaction can be done on top of a distributed WAL.
Databend-meta can be the WAL service but require a new feature that generate sequential number for key as the distributed WAL log-id.
E.g., the RPC API to append a log entry to this WAL service may look like:

append_wal(prefix: &str, payload: &[u8]) -> LogIndex;
purge_wal(prefix: &str, upto: LogIndex) -> (LogIndex, LogIndex);

• append() will internally store the payload in a key <prefix><log_index:010>(e.g. append_wal('a/', ...) stores a key 'a/0000000001', where log_index is increased by 1 for every log entry under a prefix.

• purge_wal() will delete upto the specified log index, except it will never delete the last one. Otherwise the next log_index will revert to 0. It returns

Features and limitations

• The keys <prefix><log_index> can also be accessed with KVApi. E.g., list logs for applying them to a state-machine.
• Using KVApi::upsert() on a WAL log record is illegal, it will break consistency.
• Once a WAL is created, there has to be at least one record in it. Otherwise the next log_index will revert to 0.
• This WAL does not define how to apply logs to some state-machine. It's application specific.