NodeFSStorageAdapter does not preload any document

[Daru13]

I am developing a server that stores the current state of multiple repositories on the disk using NodeFSStorageAdapter and exposes their content, i.e., the documents it contains, to its clients. This is working great as long as the documents are created while the server is running. However, whenever I stop the server and start it again, the server seems to be unaware of the locally available content of the repositories: no existing document seems to be (pre)loaded by NodeFSStorageAdapter unless I manually request them using Repo.find. This, in turn, requires knowing the IDs of all the document of the repository that exist locally. Yet, I did not find any good way to get that information using the current APIs of automerge-repo and the storage adapter I'm using.

My current workaround is to make some assumptions on how NodeFSStorageAdapter stores the data on the disk and to go through the two-level hierarchy of directories it uses to encode the document IDs immediately after creating the repository:

import type * as A from "@automerge/automerge-repo";
import * as fs from "node:fs";

function loadAutomergeDocumentsFromFiles(
    repo: A.Repo,
    pathToStorageDirectory: string
): void {
    // To retrieve all the documents IDs, we assume that NodeFSStorageAdapter
    // stores them in the given root directory under the following structure:
    // 
    // <root directory>
    //   └ <first two characters of the ID>         (1st level directories below)
    //       └ <remaining characters of the ID>     (2nd level directories below)
    //           └ ...
    // 
    // Moreover, all repositories written on disk with this utility seem to
    // store an entry called "storage-adapter-id" (under st/orage-adapter-id),
    // which must therefore NOT be treated as a document ID.

    const automergeDocumentIds = new Set<string>();

    const firstLevelDirectoryNames = fs.readdirSync(pathToStorageDirectory);
    for (let firstLevelDirectoryName of firstLevelDirectoryNames) {
        const secondLevelDirectoryNames = fs.readdirSync(`${pathToStorageDirectory}/${firstLevelDirectoryName}`);
        for (let secondLevelDirectoryName of secondLevelDirectoryNames) {
            automergeDocumentIds.add(`${firstLevelDirectoryName}${secondLevelDirectoryName}`);
        }
    }

    automergeDocumentIds.delete("storage-adapter-id");

    for (let id of automergeDocumentIds) {
        try {
            repo.find(id as A.DocumentId);
        }
        catch (error) {
            console.warn(`Error while preloading Automerge document ${id}.`);
        }
    }
}

...but it feels a bit hacky, since this code is relying on implementation details.

It would be great to have an "official" way to automatically preload all the documents of a repository that are locally available, or, at the very least, to get the list of IDs of all the documents that are stored on the disk.

For example, I imagine that it could take the form of a flag passed to the constructor of NodeFSStorageAdapter, and/or of a new (possibly optional) method to preload documents in the StorageAdapterInterface interface (which may, in turn, be exposed as a flag when creating a repository, regardless of the storage adapter that is actually being used).

Has anyone else faced a similar problem/found a better solution? If we reach an agreement on a solution, I can look into it and create a PR :).

[pvh]

Hi @Daru13. It is by design that you cannot (supportedly) introspect the contents of a repository. You should assume that other applications or users share a storage adapter. The recommended approach is to create some kind a directory document that is your "root folder" and store that one ID somewhere convenient. From there, you can relatively easily load documents recursively.

This is an important design feature in automerge-repo, because in a world where applications are always changing and different elements of your system might introduce new document types a design that assumes a homogenous repository becomes a liability.

[Daru13]

Hi @pvh, and thank you for the explanation of this design rationale.

I am currently using a dedicated repository on the disk for each of the server's Automerge repositories, so I suppose that I can safely assume that no other application/user will alter its content. That being said, I agree with you that my current solution is definitely not very future proof.

I initially wanted to try using Automerge to store all the server's data, including some local configuration data, to avoid using another form of storage, but I suppose that I will have to if I need to store the ID of some "root" document, as you suggest.

[zicklag]

What you might be able to do is take your automerge storage adapter, NodeFSStorageAdapter, in this case, and wrap it with a namespaced storage adapter, so that you know automerge's data will be stored in a specific namespace:

/** Takes a storage adapter and creates a sub-adapter by with the given namespace. */
export function namespacedSubstorage(
  storage: StorageInterface,
  ...namespaces: string[]
): StorageInterface {
  return {
    load(key) {
      return storage.load([...namespaces, ...key]);
    },
    async loadRange(key) {
      const result = await storage.loadRange([...namespaces, ...key]);
      return result.map((x) => ({
        key: x.key.slice(namespaces.length),
        data: x.data,
      }));
    },
    remove(key) {
      return storage.remove([...namespaces, ...key]);
    },
    removeRange(key) {
      return storage.removeRange([...namespaces, ...key]);
    },
    save(key, value) {
      return storage.save([...namespaces, ...key], value);
    },
  };
}

Then what you can do is you can create the automerge storage adapter like this:

const automergeStorage = namespacedSubstorge(myNodeFsStorage, "automerge")
const appStorage = namespacedSubstorge(myNodeFsStorage, "my_app")

Then you can use the automerge storage interface functions like appStorage.save("data", data), etc. and know that it won't overlap with Automerge's data, and you won't have to use a different kind of storage for your app.

[pvh]

Hi @pvh, and thank you for the explanation of this design rationale.

I am currently using a dedicated repository on the disk for each of the server's Automerge repositories, so I suppose that I can safely assume that no other application/user will alter its content. That being said, I agree with you that my current solution is definitely not very future proof.

I initially wanted to try using Automerge to store all the server's data, including some local configuration data, to avoid using another form of storage, but I suppose that I will have to if I need to store the ID of some "root" document, as you suggest.

Just a note -- apologies for the terrible lateness of it -- but we promise that no two documents will ever have the same ID so it is indeed safe to store different users' data in the same storage adapter, provided a user's data is crawlable from some root.

One thing that might be useful would be repo.gc() in the style of git gc to .delete() un-referred-to files, though it's debatable whether this should include gc-ing documents that were formerly linked as this could break history traversal.