Skip to content

WIP: Upgrade NostrDB + new experimental architecture approaching local relay model #2912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 213 commits into from

Conversation

danieldaquino
Copy link
Collaborator

Summary

⚠️ Very very WIP. Experimental. Multiple issues. Do not merge.

This is an experimental branch containing the latest NostrDB update branch: #2121
and the new architecture approaching the local relay model #2911.

This branch approaches the local relay model. To implement the local relay model (or a first prototype of it), we would need a patch similar to the following:

diff --git a/damus/Models/NostrNetworkManager/NostrNetworkManager.swift b/damus/Models/NostrNetworkManager/NostrNetworkManager.swift
index 1ce18479..9e3ab41c 100644
--- a/damus/Models/NostrNetworkManager/NostrNetworkManager.swift
+++ b/damus/Models/NostrNetworkManager/NostrNetworkManager.swift
@@ -56,7 +56,27 @@ actor NostrNetworkManager {
     
     /// Subscribes to data from Nostr
     func subscribe(filters: [NostrFilter], to: [RelayURL]? = nil) -> AsyncStream<RelayPool.StreamItem> {
-        return self.pool.subscribe(filters: filters, to: to)
+        return AsyncStream<RelayPool.StreamItem> { continuation in
+            let ndbStreamTask = Task {
+                for await ndbEvent in ndb.query(filters) {
+                    continuation.yield(.event(ndbEvent))
+                }
+                continuation.yield(.eose)
+                for await ndbEvent in ndb.stream(filters) {
+                    continuation.yield(ndbEvent)
+                }
+            }
+            let networkStreamTask = Task {
+                for await item in pool.subscribe(filters) {
+                    guard case .event(let event) = item else { continue }
+                    ndb.ingest(event)
+                }
+            }
+            continuation.onTermination = {
+                ndbStreamTask.cancel()
+                networkStreamTask.cancel()
+            }
+        }
     }
     
     func query(filters: [NostrFilter], to: [RelayURL]? = nil) async -> [NostrEvent] {

However, to apply this patch, we still need to create the interface that will allow us to use ndb_query and ndb_subscribe using the same types we use on Swift (e.g. NostrFilter, NdbNote, etc) — which is non-trivial.

Checklist

  • I have read (or I am familiar with) the Contribution Guidelines
  • I have tested the changes in this PR
  • I have opened or referred to an existing github issue related to this change.
  • My PR is either small, or I have split it into smaller logical commits that are easier to review
  • I have added the signoff line to all my commits. See Signing off your work
  • I have added appropriate changelog entries for the changes in this PR. See Adding changelog entries
    • I do not need to add a changelog entry. Reason: [Please provide a reason]
  • I have added appropriate Closes: or Fixes: tags in the commit messages wherever applicable, or made sure those are not needed. See Submitting patches

Test report

Please provide a test report for the changes in this PR. You can use the template below, but feel free to modify it as needed.

Device: [Please specify the device you used for testing]

iOS: [Please specify the iOS version you used for testing]

Damus: [Please specify the Damus version or commit hash you used for testing]

Setup: [Please provide a brief description of the setup you used for testing, if applicable]

Steps: [Please provide a list of steps you took to test the changes in this PR]

Results:

  • PASS
  • Partial PASS
    • Details: [Please provide details of the partial pass]

Other notes

[Please provide any other information that you think is relevant to this PR.]

jb55 and others added 30 commits February 13, 2025 13:08
When trying to build from rust

Signed-off-by: William Casarin <[email protected]>
Closes: damus-io/nostrdb#21
Signed-off-by: William Casarin <[email protected]>
Right now it's accidently hardcoded.

Fixes: 8376e5bca05c ("add "import -"")
Signed-off-by: William Casarin <[email protected]>
everything will be in here soon
rust doesn't like packed structures, so hide this from bindgen

This also buttons up the API so less things are exposed which is good.

Signed-off-by: William Casarin <[email protected]>
This is the start of our rust library for nostrdb. Implement idiomatic
interfaces for Ndb and NdbConfig.

Changelog-Added: Add initial rust library
Signed-off-by: William Casarin <[email protected]>
otherwise build fails

Signed-off-by: William Casarin <[email protected]>
since it was overwritten when we synced with damus

Signed-off-by: William Casarin <[email protected]>
we will be applying a patch here as well
A lot of this was pulled from core-lightning. Not sure what is actually
needed or not.

Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
since I keep overwriting it by accident

Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
We will be storing raw nostr bech32 buffers directly into nostrdb, so
adapt our bech32 code to reflect this.

When doing our content parsing pass, we will only look for strings and we
won't allocate any intermediate buffers. Only when we write this string
block to nostrdb will we actually allocate in our nostrdb output buffer
(no mallocs!)

Signed-off-by: William Casarin <[email protected]>
This is the same as cursor_slice except we don't memset afterwards

Signed-off-by: William Casarin <[email protected]>
This is a varint helper that doesn't pull larger than uint32

Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
jb55 and others added 26 commits February 13, 2025 14:43
No functional changes, just formatting cleanups

Signed-off-by: William Casarin <[email protected]>
Add a helper for sorting search words from largest to smallest. This
should help search performance. For example, let's say our search index
is like so:

"the pokemon is cool"

the
the
the
...
* 1000

Our root word search would have to start 1000 new recursive queries. By
sorting by the largest word:

pokemon
pokemon
pokemon
...
* 10

We only have to do 10 recursive searches, assuming larger words are less
common, which will likely be the case most of the time

Signed-off-by: William Casarin <[email protected]>
Update fulltext search queries to include an optional filter. This can
be used to narrow down the fulltext search. This is another step towards
nip50 support in nostrdb.

I noticed the code was exiting dubiously in certain situations... so we
fix that as well. It's possible we were missing search results because
of this.

Signed-off-by: William Casarin <[email protected]>
This adds support for nip50 fulltext searches. This allows you to use
the nostrdb query interface for executing fulltext searches instead of
the typical `ndb_text_search` api. The benefits of this include a
standardized query interface that also further filters on other fields
in the filter.

Changelog-Added: Add nip50 search filters and queries
Signed-off-by: William Casarin <[email protected]>
Add support for type KIND for bech32-encoded entities naddr and nevent
as specified in NIP-19.

Co-authored-by: kernelkind <[email protected]>
Signed-off-by: William Casarin <[email protected]>
some binding dir stoppers, and configurator
doesn't need to create a copy of the id

Signed-off-by: William Casarin <[email protected]>
This is still kind of broken until queries are switched over to nostrdb.
Will do this next

Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: Daniel D’Aquino <[email protected]>
Signed-off-by: Daniel D’Aquino <[email protected]>
Signed-off-by: Daniel D’Aquino <[email protected]>
@danieldaquino danieldaquino changed the title WIP: New NostrDB + new experimental layer separating relay pool for local relay model WIP: Upgrade NostrDB + new experimental architecture approaching local relay model Mar 6, 2025
@jb55
Copy link
Collaborator

jb55 commented Mar 19, 2025

we should get nostrdb updated and merged first before attempting this IMO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants