tikv: store WAL and Level 0 on SSD, offload higher levels to object store such as S3

[auntyellow]

Feature Request

Log data such as transaction details, browse history, may reach many TB or PB, and they must be query online. Store them on S3 may be much cheaper than SSD. Although S3 has much higher latency than SSD, it is acceptable if people just search their logs.

Consider RocksDB, where TiKV is based on: Is it possible that level 0 WAL files still be written into SSD, while level 1 and higher level SST files written into S3?

An open source project RocksDB-Cloud can set up RocksDB on S3, but there is no such an SQL database support it.

[coocood]

Sure, not yet.
But there is an experimental project unistore is developing this feature, it is not production-ready yet.

[ghost]

I have updated the title to be a little bit clearer. This feature request is for TiKV to offload Levels N and higher to an object store. It will likely be independent of TiDB.