Update to latest nostrdb #3191
Closed
Update to latest nostrdb #3191
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
making more things configurable if you have memory constraints Signed-off-by: William Casarin <[email protected]>
Add relay indexing for existing notes This patch introduces a relay index for new notes and notes that have already been stored, allowing the database to track additional relay sources for a given note. Changes: - Added `NDB_WRITER_NOTE_RELAY` to handle relay indexing separately from new note ingestion. - Implemented `ndb_write_note_relay()` and `ndb_write_note_relay_kind_index()` to store relay URLs. - Modified `ndb_ingester_process_event()` to check for existing notes and append relay info if necessary. - Introduced `ndb_note_has_relay()` to prevent duplicate relay entries. - Updated LMDB schema with `NDB_DB_NOTE_RELAYS` (note_id -> relay) and `NDB_DB_NOTE_RELAY_KIND` (relay + kind + created_at -> note). - Refactored `ndb_process_event()` to use `ndb_ingest_meta` for tracking relay sources. - Ensured proper memory management for relay strings in writer thread. With this change, nostrdb can better track where notes are seen across different relays, improving query capabilities for relay-based data retrieval. Signed-off-by: William Casarin <[email protected]>
This is a simple cursor that walks the NDB_DB_NOTE_RELAYS db Signed-off-by: William Casarin <[email protected]>
for debugging Signed-off-by: William Casarin <[email protected]>
This fixes a race condition where if multiple of the same note is processed at the same time, we still manage to write the note relays Signed-off-by: William Casarin <[email protected]>
There were a few race conditions and lmdb bugs in the relay index implementation. Fix those! Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Add support for relay-based filtering in nostr queries.
Filters can now include a "relays" field. Optimal performance when
you include a kind as well:
{"relays":["wss://pyramid.fiatjaf.com/"], "kinds":[1]}
This corresponds to a `ndb` query like so:
$ ndb query -r wss://pyramid.fiatjaf.com/ -k 1 -l 1
using filter '{"relays":["wss://pyramid.fiatjaf.com/"],"kinds":[1],"limit":1}'
1 results in 0.094929 ms
{"id":"277dd4ed26d0b44576..}
Signed-off-by: William Casarin <[email protected]>
This should help author kind query performance
The basic idea of this is to allow you to use the standard
nip50 query interface to search for profiles using our profile
index.
query: {"search":"jb55", "kinds":[0]}
will result in a profile_search query plan that searches kind0 profiles
for the corresponding `name` or `display_name`.
Signed-off-by: William Casarin <[email protected]>
This adds some helpers for adding custom filtering logic to nostr filters. These are just a callback and a closure. There can only be one custom callback filter per filter. Fixes: damus-io/nostrdb#33 Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
These queries are broken anyways. Rely on scans until we fix this Signed-off-by: William Casarin <[email protected]>
Rogue relays could in theory attack nostrdb by replaying ids and signatures from other notes. This fixes this weakness by calculating the id again in ndb_note_verify. There is no known relays exploiting this, but lets get ahead of it before we switch to the outbox model in damus iOS/notedeck Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
this is causing heap corruption on the windows build This reverts commit a8d6925a5b33ddbdd4306423527b5d8314f7dd36.
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
before we weren't checking this, meaning we were getting results from other keys. oops. Reported-by: Jeff Gardner Fixes: #84 Signed-off-by: William Casarin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should be breaking as I've changed the ndb_note_verify interface, but throwing this up here so I don't forget