Skip to content

Update to latest nostrdb #2121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 213 commits into
base: master
Choose a base branch
from
Open

Update to latest nostrdb #2121

wants to merge 213 commits into from

Conversation

jb55
Copy link
Collaborator

@jb55 jb55 commented Apr 4, 2024

This branch updates nostrdb to the latest version

Things done and left to do:

  • Update nostrdb
  • Switch to nostrdb for note block rendering
    - [ ] Switch to local relay for querying and subscriptions saving this for later
  • Fix DM rendering

What's new in this version:

  • full query support (swift interface for this is WIP)
  • nip50 search support: full text queries on filters (profile search, etc)

@jb55 jb55 linked an issue Apr 4, 2024 that may be closed by this pull request
@jb55 jb55 force-pushed the nostrdb-update branch from a3d9cb3 to 3a6dcdc Compare April 4, 2024 22:34
@jb55
Copy link
Collaborator Author

jb55 commented Apr 5, 2024

This broke a bunch of tests which I'm fixing now

@jb55
Copy link
Collaborator Author

jb55 commented Apr 5, 2024

One thing I did here is add nostrdb's patch history to our local copy of nostrdb. This ensure we don't lose bisectability when updating nostrdb. I've also removed a lot of overlap between damus' C code and nostrdb, so most of the C code should be in nostrdb now.

@jb55 jb55 linked an issue Apr 8, 2024 that may be closed by this pull request
@jb55 jb55 force-pushed the nostrdb-update branch 3 times, most recently from 10bc4f4 to c476272 Compare April 25, 2024 22:50
This was linked to issues May 6, 2024
@jb55 jb55 force-pushed the nostrdb-update branch 3 times, most recently from 1f3d846 to f8931a6 Compare September 1, 2024 12:00
@danieldaquino danieldaquino mentioned this pull request Sep 26, 2024
12 tasks
@jb55 jb55 mentioned this pull request Oct 14, 2024
@jb55 jb55 force-pushed the nostrdb-update branch 3 times, most recently from 8bc95f2 to e56f0a5 Compare February 14, 2025 00:26
@jb55 jb55 marked this pull request as ready for review February 14, 2025 00:30
@jb55
Copy link
Collaborator Author

jb55 commented Feb 14, 2025

@danieldaquino this is ready for testing!

@jb55 jb55 changed the title Update nostrdb, switch to local relay model Update to latest nostrdb Feb 14, 2025
@jb55
Copy link
Collaborator Author

jb55 commented Feb 14, 2025

looks like there are a fix issues with block parsing...

@jb55
Copy link
Collaborator Author

jb55 commented Feb 17, 2025

Fixed an iOS crash in the latest version of nostrdb. is a bus error (unaligned memory access) when trying to parse an empty json string when processing inner kind6 reposts (this is a new feature in this version of nostrdb)

jb55 and others added 2 commits June 9, 2025 17:57
When trying to build from rust

Signed-off-by: William Casarin <[email protected]>
Closes: damus-io/nostrdb#21
Signed-off-by: William Casarin <[email protected]>
jb55 and others added 7 commits June 9, 2025 17:57
No functional changes, just formatting cleanups

Signed-off-by: William Casarin <[email protected]>
Add a helper for sorting search words from largest to smallest. This
should help search performance. For example, let's say our search index
is like so:

"the pokemon is cool"

the
the
the
...
* 1000

Our root word search would have to start 1000 new recursive queries. By
sorting by the largest word:

pokemon
pokemon
pokemon
...
* 10

We only have to do 10 recursive searches, assuming larger words are less
common, which will likely be the case most of the time

Signed-off-by: William Casarin <[email protected]>
Update fulltext search queries to include an optional filter. This can
be used to narrow down the fulltext search. This is another step towards
nip50 support in nostrdb.

I noticed the code was exiting dubiously in certain situations... so we
fix that as well. It's possible we were missing search results because
of this.

Signed-off-by: William Casarin <[email protected]>
This adds support for nip50 fulltext searches. This allows you to use
the nostrdb query interface for executing fulltext searches instead of
the typical `ndb_text_search` api. The benefits of this include a
standardized query interface that also further filters on other fields
in the filter.

Changelog-Added: Add nip50 search filters and queries
Signed-off-by: William Casarin <[email protected]>
Add support for type KIND for bech32-encoded entities naddr and nevent
as specified in NIP-19.

Co-authored-by: kernelkind <[email protected]>
Signed-off-by: William Casarin <[email protected]>
@jb55 jb55 force-pushed the nostrdb-update branch from 54e044a to c59ffda Compare June 12, 2025 16:24
@jb55
Copy link
Collaborator Author

jb55 commented Jun 12, 2025

The last issue seems to be DM rendering

danieldaquino and others added 16 commits June 12, 2025 09:36
The Address Sanitizer detected a heap buffer overflow during a memcpy operation
in nostrdb.c associated with note parsing.

It was found that not enough memory was being allocated to the buffer to
support all the content parsing.

Allocation size was increased to support the memory needed for the
parsing operations. However, the new number was not carefully calculated
as we will not run into this code path once we switch to the local relay
model.

Changelog-Fixed: Fixed memory error in nostrdb
Signed-off-by: Daniel D’Aquino <[email protected]>
I wanted to not amend this since we've already applied
it in the nostrdb-update branch on damus and I don't want
to conflict

Signed-off-by: William Casarin <[email protected]>
some binding dir stoppers, and configurator
doesn't need to create a copy of the id

Signed-off-by: William Casarin <[email protected]>
This is still kind of broken until queries are switched over to nostrdb.
Will do this next

Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
Signed-off-by: William Casarin <[email protected]>
This commit introduces new interfaces for working with NostrDB from
Swift, including `NostrFilter` conversion, subscription streaming via
AsyncStreams and lookup/wait functions.

No user-facing changes.

Changelog-None
Signed-off-by: Daniel D’Aquino <[email protected]>
Closes: #2885
Changelog-Changed: Use NostrDB for rendering note contents
Signed-off-by: Daniel D’Aquino <[email protected]>
NostrDB relies on manual memory management, so it is a good idea to
enable the address sanitizer on debug configurations, as it helps find
memory-related issues on the app, which will allow us to identify memory
issues and potential crashes earlier in the development process.

Changelog-None
Signed-off-by: Daniel D’Aquino <[email protected]>
@jb55 jb55 force-pushed the nostrdb-update branch from 6626dcb to 57018b4 Compare June 12, 2025 16:38
This makes it easier to work from the command line when needed

Changelog-None
Signed-off-by: Daniel D’Aquino <[email protected]>
Some tests have been broken at some point during the nostrdb migration.
Disable them for now and address them later
(#3112)

Signed-off-by: Daniel D’Aquino <[email protected]>
Currently NostrDB does not seem to handle encryption/decryption of DMs.
Since NostrDB now controls the block parsing process and fetches note
contents directly from the database, we have to add a specific condition
that injects decrypted content directly to the ndb content parser.

This is done in conjunction with some minor refactoring to `NdbBlocks`
and associated structs, as in C those are separated between the content
string and the offsets for each block, but in Swift this is more
ergonomically represented as a standalone/self-containing object.

No changelog entry is added because the previously broken version was
never released to the public, and therefore this fix produces no
user-facing changes compared to the last released version.

Changelog-None
Closes: #3106
Signed-off-by: Daniel D’Aquino <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to nostrdb for note block rendering Local Relay Model (nostrdb) Security concern Do signature checking in the background
9 participants