Add a pubsub database implementation

[Liam-DeVoe]

TODO: write real docs for this? and figure out whether to put watchdog in test.in or tools.in?

[Zac-HD]

High-level feedback: I'm really excited to have a high-performing option for this!

Small design change to support that: we should notify only on save events (inc. move), and pass the listener the changed k:v pair(s), rather than just a "something changed" notification. That way listeners never need to rescan the database!

[Liam-DeVoe]

Not on deletion also? We should definitely pass the k:v pairs (I realized this last night), but if we want consumers to be able to losslessly reconstruct the db state without rescanning then we also need to fire deletion events right?

move is bit awkward with both a deletion and save event, but consumers (hypofuzz) are going to be debouncing anyway, so I don't think it will impact much.

[Zac-HD]

Yeah, good point. Let's make it a stream of ("save"/"delete", key, value).

Then our watcher can subscribe, scan once, and replay updates onto a cheap-to-query InMemoryExampleDatabase; and maybe even keep track of the keys for a lazily-updated derived view of deserialized contents.

[Liam-DeVoe]

hm, problem: DirectoryBasedExampleDatabase stores key paths lossily with a hash, with no way to go from hash -> key unless we track the reverse mapping in memory (which isn't complete in this case, because we might want to emit e.g. a save event for a file saved by another database accessing the same directory, where we never see the original key).

We could add a new .key_mapping json file in the directory for this? Backwards compatibility is fine, we just don't emit events we don't have a mapping for.

e: probably better to do one .key file per directory to avoid write races or rereading a large file

[Zac-HD]

idea: just have a b"keys" key, which saves the value of each known key (similar to HypoFuzz, but more general). Then we can scan that at start-listening time, track the hash->key mapping, and we'll be notified about new keys when they're written!

[Liam-DeVoe]

that works wonderfully!

One other snag: DirectoryBasedExampleDatabase cannot propagate the value which was deleted, because the file is already gone by the time we get the event. I'm currently typing the event as type: Literal["delete"], key: bytes, value: Optional[bytes] where value=None indicates we don't know what the deleted value is, with the idea being hypofuzz would rescan just that key on deletions.

[Liam-DeVoe]

This was an enormous pain to test, but on the bright side I am confident in the implementation now, and have picked up on a few minor implementation savings (move with src == dest in redis) along the way. I half expect these tests to be flaky though...

[Liam-DeVoe]

the linux details for move here are weird, but not really harmful, just redundant/inefficient. I suspect this is at the os/watchdog level but don't have a linux machine to test further. We use os.renames for db.move with a fallback to save/delete, which may explain some of the divergent behavior.

[Zac-HD]

github "codespaces" might be helpful to debug?

[Zac-HD]

Looks good, comments below are non-blocking 🙂

[Zac-HD]

github "codespaces" might be helpful to debug?