Automatically downloading copies of fdb_c dynamic library

[josephg]

The foundationdb client is currently quite difficult to deploy because it requires the user to first install the foundationdb client (or at least a local copy of libfdb_c matching the version of their database).

Now that fdb_c is on the official download page for all major systems, another option has opened up. The node bindings could:

• Accept one or many fdb server version numbers (eg 6.0.15) before trying to connect
• Download a copy of fdb_c for your OS & database version from the official website, downloading the file into some temporary / cache directory.
• Set NetworkOptions.external_client_directory to that directory
• Then connect as normal

Thanks to the prebuildify process, this would allow the fdb bindings to work without any external system configuration. You could just npm install foundationdb and then (after specifying a few versions on startup / through environment variables), you could connect directly to your fdb server. On first connect the client would download an ~8MB library file, but after that it'd be smooth sailing.

This would also make it way easier to deploy to AWS or heroku or whatever.

Anyway, if anyone wants to have a go at this, it'd be a great first issue to help out with!

[ryanworl]

How would this interact with using the multi-version client?

[josephg]

I'm imagining something like this:

await fdb.supportServers(['5.2', '6.0'])
fdb.setAPIVersion(600)
const db = await fdb.openDatabase()
// ...

If you call supportServers it'd download the requested versions of fdb_c.so/dylib/dll into some standard directory ($HOME/.fdb_c or something). Then it would set the network option external_client_directory to point to that location.

We'd need to also change how the native part of the module loads fdb_c so it can use one of the copies of fdb_c in that location. (And we'd need to verify checksums or something too I imagine).

[amv]

I'm not sure if this is far out there as an idea, but would it be possible to include a specific version of the fdb_c library as a WASM module in the sources of this library?

Another option might be to create a separate npm module from the WASM-compiled fdb_c, and release a new npm version for each fdb_c release. This would make it possible to easily depend on an another npm module to get a specific version of the fdb_c library.

Especially if the WASM compiled module of fdb_c ends up being significantly slower than the native one, having the option to choose between the native fbd_c library installed on your system and the WASM compiled library might also be a good idea, but I'm not sure what would be the best way to implement it.

[josephg]

I'm not sure if this is far out there as an idea, but would it be possible to include a specific version of the fdb_c library as a WASM module in the sources of this library?

Sadly no - the version of the local fdb_c library needs to exactly match the version of the foundationdb server you're connecting to. We could ship a copy of fdb_c for every OS/version combination - but at that point its getting a bit silly. Hence the idea of just downloading libfdb_c on demand.

I also haven't been able to get fdb_c to compile with wasm. You would need to wire through the networking calls, and I think the client library uses a coroutine library that is written in ASM. I'm not sure if it would be possible to port that to wasm.

That said, work is apparently progressing on making a standard fdb protocol. Commentary here. If / when this lands we'll aim for a native javascript implementation of the protocol and replace the default backend of this library with that.

[amv]

Hmm.. The RCP gateway sounds like a good layer (or a scalable core component) to add to fdb, and it would bring various benefits (as discussed behind the links).. But at least to me a library talking through a RCP gateway sounds like a very different animal than a library that uses libfdb_c to coordinate among the cluster components.

I understand that if the opportunity arises (gRPC gateway layer becomes standard with fdb clusters), it would be much easier to maintain the gRPC based version than the current version, and it would probably cater to a vast majority of the use cases that a node library user would have.

However I do see that there might also be a need to use the model where the node processes are talking directly to the cluster via libfdb_c. Maybe those that need it can fork their own versions of the lib though, and this lib can go towards the gRPC gateway scenario. Given that the need for this other lib would probably be in the margins (given the gRPC route exists), maybe this dependency lib installation automation issue is not of the highest priority right now.

[josephg]

But at least to me a library talking through a RCP gateway sounds like a very different animal than a library that uses libfdb_c to coordinate among the cluster components.

Other than performance, how do you imagine it being different in practice? I mean, you might have to turn on the gRPC gateway but I imagine the specs will be similar.

Anyway, we could still implement the proposal I wrote up above if that would be useful for some folks. I imagine the grpc gateway stuff will be a ways out anyway.

[amv]

Not sure how much it will actually end up showing in usage, but I guess at least the failure model and need for timeouts will be somewhat different when the gRPC server can disappear, get switched, and experience lag or transport errors at any point in the execution (in addition to all the watches, timeouts, native exceptions and error callbacks being somehow packaged in the wire protocol).

But given the current situation, I think your proposal is good as long as it

• A) provides a hook which allows doing the download at instance warmup stage

• B) provides an option to download/copy the lib source from whichever location you choose (and defaults to the public website download url), and

• C) adds a step which verifies the sha256 lib file checksum before proceeding with the execution.

Heroku user would probably package the .so tar file (or it's contents) in their app bundles, so the possibility to choose a local file in addition to an url (and maybe disabling the verification step with local files) would be of benefit.

I think the requirement C would require the library to contain the checksum data of known versions within the package (which is much more sane than bundling all of them) and it should be possible to manually add urls and checksums for situations where the npm lib can't be updated, but one wishes to try and use a newer version of the fdb_c lib, which is supposed to be backwards compatible.