Race condition in `directory.createOrOpen`

[aikoven]

When attempting to call directory.createOrOpen for a non-existent directory multiple times in a single transaction in parallel, two different subspaces are created. Only one of them gets referenced by the directory path. The other one becomes orphan.

Reproduction code:

const foundationdb = open();

const [dir1, dir2] = await foundationdb.doTransaction(tn =>
  Promise.all([
    directory.createOrOpen(tn, ['test']),
    directory.createOrOpen(tn, ['test']),
  ]),
);

expect(dir1.content.prefix).toEqual(dir2.content.prefix);  // fails

[keijokapp]

The official Python bindings work the same way:

import fdb, gevent

fdb.api_version(730)
db = fdb.open(event_model='gevent')
db.clear_range(b'', b'\xff')

tr = db.create_transaction()
greenlets = [
  gevent.spawn(lambda: fdb.directory.create_or_open(tr, (b'test',))),
  gevent.spawn(lambda: fdb.directory.create_or_open(tr, (b'test',)))
]
print([greenlet.value.rawPrefix for greenlet in gevent.wait(objects=greenlets)])

# [b'\x15\x13', b'\x15=']

(As expected, the returned directory prefixes are the same when calling create_or_open with db instead of tr)

This is rather disappointing TBH. FoundationDB is designed to coordinate transactions in multi-datacenter scale applications but doesn't coordinate their own library behavior within a single-threaded transaction. This is not surprising though. All layers I've looked into (and layers I've developed myself) struggle with intra-transaction concurrency handling. They either neglect it (as in this case), calling it an undefined behavior; or use overly pessimistic locking which has its own issues.

In an ideal world, FoundationDB would provide a first-class support for nested/composable client-side transactions, so that the layer and application developers would never have to worry about these things. FDB transaction state is managed client-side anyway, so a basic solution doesn't need any server-side support. I've been on-and-off working on such implementation for NodeJS but it's far from being usable. It aims to be an emulation of FDB transactions over an arbitrary data backend (eg. other FDB transactions).

Anyway, I don't know what's better solution for now: locks or UB. I haven't used the directory layer so I don't know what problems locks may cause. UB is also nasty but that's what official client libraries (at least the python one) do.

[aikoven]

Thank you for elaborate comment @keijokapp!

Yeah, I expected that there's just a lack of concurrency safety in the layer implementation. For now I just wrapped my directory-related code with locks. Which was rather easy, however the original problem was not obvious at all and lead to the need to manually clear orphan directories (and some data loss).

I like your idea with nested transactions!