Skip to content

Add native NIP-77 negentropy sync for efficient timeline updates #3444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alltheseas wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add native NIP-77 negentropy sync for efficient timeline updates #3444

alltheseas wants to merge 5 commits into from

Conversation

@alltheseas
Copy link
Collaborator

@alltheseas alltheseas commented Dec 17, 2025 •
edited
Loading

Summary

Implements NIP-77 negentropy set reconciliation for efficient timeline sync. Instead of re-fetching thousands of events on app foreground, negentropy efficiently identifies only the events we're missing by comparing fingerprints of our local event set against the relay's set.

Key features:

  • Native C implementation integrated with nostrdb for zero-copy event fingerprinting
  • Swift wrapper with async/await API
  • Caches relay NIP-77 support to avoid repeated NIP-11 checks
  • Two-phase timeout handling for multi-round reconciliation
  • Detects "negentropy disabled" NOTICE for relays that advertise but don't support NIP-77

Performance: Reduces network traffic from 5000+ events to ~50-200 missing events on typical foreground sync.

Checklist

Standard PR Checklist

  • I have read (or I am familiar with) the Contribution Guidelines
  • I have tested the changes in this PR
  • I have profiled the changes to ensure there are no performance regressions, or I do not need to profile the changes.
    • Negentropy sync runs async in background, does not block UI. Tested with 50k events in nostrdb.
  • I have opened or referred to an existing github issue related to this change.
  • My PR is either small, or I have split it into smaller logical commits that are easier to review
  • I have added the signoff line to all my commits. See Signing off your work
  • I have added appropriate changelog entries for the changes in this PR. See Adding changelog entries
    • I do not need to add a changelog entry. Reason: Feature not yet user-visible, will add changelog when enabling in UI
  • I have added appropriate Closes: or Fixes: tags in the commit messages wherever applicable, or made sure those are not needed. See Submitting patches

Test report

Device: iPhone 16 Pro Simulator

iOS: 18.3.1

Damus: f7d3c8c

Setup: Account with ~50,000 events in local nostrdb, connected to relay.damus.io and pyramid.fiatjaf.com

Steps:

  1. Launch app (triggers negentropy sync on foreground)
  2. Observe logs for sync completion
  3. Background and foreground app to test reconnect sync
  4. Verify nos.lol (advertises NIP-77 but disabled) is detected and cached as unsupported

Results:

  • PASS
    • relay.damus.io: 14 reconciliation rounds, synced ~22k missing events
    • pyramid.fiatjaf.com: 4 reconciliation rounds, synced ~275 missing events
    • nos.lol: Correctly detected as disabled via NOTICE, cached as unsupported

Other notes

Closes: #3269
Builds off damus-io/nostrdb#103

🤖 Generated with Claude Code

@alltheseas
Add ndb_negentropy C implementation for NIP-77
635842b 
Native negentropy set reconciliation for NostrDB, implementing the
NIP-77 protocol for efficient event synchronization with relays.

Features:
- MSB-first varint encoding/decoding
- Range encoding/decoding (SKIP, FINGERPRINT, IDLIST, IDLIST_RESPONSE)
- Storage management with sorted (timestamp, id) pairs
- Reconciliation state machine with have/need ID tracking
- Support for relay.damus.io (MAX_RANGES=8192 for large messages)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Signed-off-by: alltheseas
@alltheseas
Add NIP-77 negentropy sync using native nostrdb
b04c69c 
Swift implementation of NIP-77 set reconciliation for efficient
event synchronization with relays.

Components:
- NdbNegentropy.swift: Swift bindings for ndb_negentropy C API
- NegentropySync.swift: Sync coordinator with NIP-11 detection
- NostrResponse: NEG-MSG, NEG-ERR, CLOSED message parsing
- RelayConnection/Pool: Negentropy message handling

Configuration tuned for relay.damus.io:
- 1MB response buffer (responses can exceed 590KB)
- splitCount=4 for smaller response sizes
- frameSizeLimit=32KB

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Signed-off-by: alltheseas
@alltheseas
Add unit tests for nostrdb negentropy
0c5cd98 
14 tests covering:
- Storage initialization, sealing, item addition
- Reconciliation context setup
- Initial message generation (protocol version 0x61)
- Invalid message handling
- Memory management (storage deallocation, reference counting)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Signed-off-by: alltheseas
@alltheseas
Optimize negentropy sync with caching and improved reliability
f7d3c8c 
Improvements to negentropy sync performance and reliability:

Startup optimization:
- Cache relay NIP-77 support status to avoid NIP-11 checks on every startup
- Sync known-supported relays immediately, check unknown relays in background
- Add settling delay (1.5s) before sync to let connections stabilize

Timeout handling:
- Two-phase timeout: 10s for first response, 30s inactivity between messages
- Per-relay sync tracking instead of global lock (allows concurrent syncs)
- Add debug logging for reconciliation failures

Reliability fixes:
- Detect "negentropy disabled" NOTICE and mark relay unsupported immediately
- Fix NOTICE parsing to capture message text
- Fix duplicate sync race condition
- Only trigger sync on reconnect (initial connects use background NIP-11 check)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Signed-off-by: alltheseas
Closes: damus-io#3269
@alltheseas alltheseas mentioned this pull request Dec 17, 2025
Open
@alltheseas
Copy link
Collaborator Author

alltheseas commented Dec 17, 2025

You can search for negentropy on xcode console

image

@alltheseas alltheseas mentioned this pull request Dec 17, 2025
Open
3 tasks
@danieldaquino danieldaquino added the pr-in-queue This PR is waiting in a queue behind their other PRs marked with the label `pr-active-review`. label Dec 20, 2025
@alltheseas alltheseas mentioned this pull request Dec 23, 2025
Draft
9 tasks
@danieldaquino danieldaquino added pr-active-review This PR is actively being reviewed, or is next on the list to be reviewed. pr-in-queue This PR is waiting in a queue behind their other PRs marked with the label `pr-active-review`. and removed pr-in-queue This PR is waiting in a queue behind their other PRs marked with the label `pr-active-review`. pr-active-review This PR is actively being reviewed, or is next on the list to be reviewed. labels Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jb55 jb55 Awaiting requested review from jb55

@danieldaquino danieldaquino Awaiting requested review from danieldaquino

Assignees

No one assigned

Labels

pr-in-queue This PR is waiting in a queue behind their other PRs marked with the label `pr-active-review`. technical

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement negentropy optimization

2 participants

@alltheseas @danieldaquino