[bug] Oxia write ops out-of-ordering causes data loss

[dao-jun]

In the case of: Put -> Delete -> Put, the oxia server may cause data loss,
you can reproduce the issue by the test:

func TestPutThenDeleteThenPut(t *testing.T) {
	standaloneServer, err := NewStandalone(NewTestConfig(t.TempDir()))
	assert.NoError(t, err)
	defer standaloneServer.Close()

	client, err := oxia.NewAsyncClient(standaloneServer.ServiceAddr(), oxia.WithBatchLinger(1*time.Second), oxia.WithMaxRequestsPerBatch(1000))
	assert.NoError(t, err)

	defer client.Close()

	putResult := client.Put("test-key", []byte("test-value"))
	deleteResult := client.Delete("test-key")
	putResult2 := client.Put("test-key", []byte("test-value2"))

	r1 := <-putResult
	r2 := <-deleteResult
	r3 := <-putResult2

	assert.NoError(t, r1.Err)
	assert.NoError(t, r2)
	assert.NoError(t, r3.Err)

	v := <-client.Get("test-key")
	assert.NoError(t, v.Err)

	assert.Equal(t, []byte("test-value2"), v.Value)
}

This is because of the write ops in WriteRequest send to server is out of ordering, the oxia client will split Put -> Delete -> Put into
2 batches: [put, put], [delete]

[mattisonchao]

Hi @dao-jun

We will not provide guarantees for concurrent operation. If you need to handle semantics after the operation, you should wait for the previous operation to finish.

[merlimat]

To add to @mattisonchao answer. If you need to ensure the state of the record for an operation, you should be using a condition in the put/delete operations

[dao-jun]

This is a clearly problematic scenario, and we can easily fix it without introducing additional overhead or breaking changes. Why not consider fixing it?

If even this semantics cannot be satisfied, what level of consistency guarantee can Oxia provide?

In Pulsar, MetadataStore provides Put and Delete methods, did we using a condition to protect it?

Does ETCD and Zookeeper also not guarantee the write ops ordering?