feat(sds): persistent storage for history #2741

danisharora099 · 2025-11-14T21:14:53Z

Problem / Description

Message history is currently stored only in memory, causing SDS to start recovery afresh every time.

Solution

This PR introduces persistent message history that survives application restarts using localStorage (browser), with an option to provide custom storage providers.

Changes

MemLocalHistory now optionally sets up Storage
Storage abstraction with automatic localStorage usage in browsers with serialisation support
- Optionally can provide custom Storage providers
Backwards compatible MessageChannel with persistent storage support, by default
New tests

Notes

Persistent storage by default (if available)
- Can opt out of persistent storage
- Non-browser environments require custom storage provider, otherwise memory-only
Supports both browsers and NodeJS
Backwards compatibility maintained

Checklist

Code changes are covered by unit tests
Code changes are covered by e2e tests, if applicable
Dogfooding has been performed, if feasible
A test version has been published, if required
All CI checks pass successfully

packages/sds/src/message_channel/message_channel.ts

packages/sds/src/message_channel/persistent_history.ts

fryorcraken · 2025-11-15T00:10:18Z

packages/sds/src/message_channel/persistent_history.ts

+ *
+ * If no storage backend is available, this behaves like {@link MemLocalHistory}.
+ */
+export class PersistentHistory extends MemLocalHistory {


Why are you extending another class instead of the interface? Avoid abstractions as a rule of thumb

extended so that we could get the implementations for the following functions without needing to reimplement:

push(...items: ContentMessage[]): number; some( predicate: ( value: ContentMessage, index: number, array: ContentMessage[] ) => unknown, thisArg?: any ): boolean; slice(start?: number, end?: number): ContentMessage[]; find( predicate: ( value: ContentMessage, index: number, obj: ContentMessage[] ) => unknown, thisArg?: any ): ContentMessage | undefined; findIndex( predicate: ( value: ContentMessage, index: number, obj: ContentMessage[] ) => unknown, thisArg?: any ): number;

Yes but you already know you want to get rid of some of that with the optimisation so I would avoid the abstraction usage.

You can also extract those logic if necessary. Saving code is not always the best way forward. See my new comments where naming is confusing.

packages/sds/src/message_channel/persistent_history.ts

fryorcraken · 2025-11-15T00:13:05Z

packages/sds/src/message_channel/persistent_history.ts

+    this.restore();
+  }
+
+  public override push(...items: ContentMessage[]): number {


Yeah, don't use abstraction. Its not worth the indirection you are bringing in

switched to composition!

packages/sds/src/message_channel/persistent_history.ts

fryorcraken · 2025-11-15T00:14:39Z

packages/sds/src/message_channel/persistent_history.ts

+  }
+
+  private persist(): void {
+    if (!this.storage) {


Hum, does it make sense for it to be constructed without storage?

we want the class to behave like MemLocalHistory if no storage is provided, right?

or perhaps, that should be handled one level above where MessageChannel chooses between MemLocalHistory and PersistentHistory -- suggestions?

we want the class to behave like MemLocalHistory if no storage is provided, right?

So you are saying that if we instantiate a persistent local history, but there is no persistence to it, then we want it to behave like memory local history? Think about the footgun you are setting up for developers.

If there is no way to persist the history, then the class handling persisting history should not be instantiable.

Indeed:

if (storage instanceof PersistentStorage) { this.storage = storage; log.info("Using explicit persistent storage"); } else if (typeof storage === "string") { this.storage = PersistentStorage.create(storage); log.info("Creating persistent storage for channel", storage); } else { this.storage = undefined; log.info("Using in-memory storage"); }

packages/sds/src/message_channel/persistent_history.ts

github-actions · 2025-11-20T21:31:44Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku node	96.31 KB (+0.08% 🔺)	2 s (+0.08% 🔺)	693 ms (+0.27% 🔺)	2.7 s
Waku Simple Light Node	147.63 KB (+0.03% 🔺)	3 s (+0.03% 🔺)	739 ms (+14.37% 🔺)	3.7 s
ECIES encryption	22.62 KB (0%)	453 ms (0%)	584 ms (+113.69% 🔺)	1.1 s
Symmetric encryption	22 KB (0%)	440 ms (0%)	457 ms (+171.08% 🔺)	897 ms
DNS discovery	52.17 KB (0%)	1.1 s (0%)	400 ms (+17.71% 🔺)	1.5 s
Peer Exchange discovery	52.91 KB (0%)	1.1 s (0%)	406 ms (+87.39% 🔺)	1.5 s
Peer Cache Discovery	46.64 KB (0%)	933 ms (0%)	398 ms (-8.35% 🔽)	1.4 s
Privacy preserving protocols	77.26 KB (-0.07% 🔽)	1.6 s (-0.07% 🔽)	338 ms (-46.59% 🔽)	1.9 s
Waku Filter	79.76 KB (-0.08% 🔽)	1.6 s (-0.08% 🔽)	662 ms (+27.57% 🔺)	2.3 s
Waku LightPush	78.06 KB (+0.11% 🔺)	1.6 s (+0.11% 🔺)	710 ms (+59.55% 🔺)	2.3 s
History retrieval protocols	83.73 KB (-0.02% 🔽)	1.7 s (-0.02% 🔽)	640 ms (-19.23% 🔽)	2.4 s
Deterministic Message Hashing	28.98 KB (0%)	580 ms (0%)	171 ms (-0.33% 🔽)	751 ms

danisharora099 · 2025-11-27T02:29:14Z

Restructured the PR:
instead of keeping PersistentHistory as a module over MemLocalHistory, now setup PersistentStorage as a separate class that can be optionally used as a storage in MemLocalHistory -- perhaps a followup will be to update the name MemLocalHistory to just History

fryorcraken · 2025-11-27T04:31:15Z

packages/sds/src/message_channel/mem_local_history.ts

+export interface ILocalHistory {
+  length: number;
+  push(...items: ContentMessage[]): number;
+  some(
+    predicate: (
+      value: ContentMessage,
+      index: number,
+      array: ContentMessage[]
+    ) => unknown,
+    thisArg?: any
+  ): boolean;
+  slice(start?: number, end?: number): ContentMessage[];
+  find(
+    predicate: (
+      value: ContentMessage,
+      index: number,
+      obj: ContentMessage[]
+    ) => unknown,
+    thisArg?: any
+  ): ContentMessage | undefined;
+  findIndex(
+    predicate: (
+      value: ContentMessage,
+      index: number,
+      obj: ContentMessage[]
+    ) => unknown,
+    thisArg?: any
+  ): number;
+}


Does not seem to be the right place to define this interface. Also, it is the same as before (Pick<Array...).

tackled here: #2745

fryorcraken

The naming is very confusing. It is unclear what is actually related to local history, vs the local storage interfaces.

Not sure how the message channel is supposed to use the persistent storage when there is no persistent storage implementing ILocalHistory.

fryorcraken · 2025-11-29T03:50:57Z

packages/sds/src/message_channel/mem_local_history.ts

 * at next push.
 */
-export class MemLocalHistory {
+export interface ILocalHistory {


define the interface alongside the class that needs this interface, aka, message channel.

It is odd to define the interface along side one of the implementations.

ILocalHistory is being implemented by LocalHistory (prev MemLocalHistory), so it needs it as well. Would you suggest moving it to MessageChannel?

Since the concept for LocalHistory belongs in this file, I believed it's good design to keep it close to the implementation. The interface exists because of the class

Based on your observation that it's only being used once, removed it.

packages/sds/src/message_channel/local_history.ts

fryorcraken · 2025-11-29T03:52:24Z

packages/sds/src/message_channel/local_history.ts

+}
+
+export type MemLocalHistoryOptions = {
+  storage?: ChannelId | PersistentStorage;


What are you doing here? Why would you use persistent storage for the Memory implementation?

Ok, no I got it, see my recent comment on the PR.

I would not have this kind of type switching here.

You can have a storagePrefix string that is applied in any case (whether you use browser localStorage or fs). It is relevant to both.

Then, for PersistentStorage, could we instead use the package.json browser feature and have 2 files:

browser-localStorage.ts
localStorage.ts

Both of them would expert a LocalStorage (what you did with PersistentStorage class, except that for the browser one, the LocalStorage class is just a thin wrap on the browser localStorage

Okay, interesting, that's probably a neater solution.

packages/sds/src/message_channel/message_channel.ts

fryorcraken · 2025-11-29T03:54:10Z

packages/sds/src/message_channel/message_channel.ts

    this.incomingBuffer = [];
-    this.localHistory = localHistory;
+    this.localHistory =
+      localHistory ?? new MemLocalHistory({ storage: channelId });


Considering that this is not an API we actually want to expose, (ReliableChannel is), then it's fine to not have default.

fryorcraken · 2025-11-29T03:54:46Z

packages/sds/src/message_channel/persistent_storage.ts

+export interface HistoryStorage {
+  getItem(key: string): string | null;
+  setItem(key: string, value: string): void;
+  removeItem(key: string): void;


what is that? it does not seem to be "History" right? It's supposed to be the interface for local storage, right? Call it ILocalStorage then

fryorcraken · 2025-11-30T23:09:51Z

After more review, I now better understand what is done here:

  ┌─────────────────────────────────────────────────────────────┐
  │                     MessageChannel                          │
  │  (defaults to MemLocalHistory with channelId for storage)   │
  └──────────────────────────┬──────────────────────────────────┘
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │                    MemLocalHistory                          │
  │  implements ILocalHistory                                   │
  │  - Manages in-memory message array                          │
  │  - Delegates persistence to PersistentStorage               │
  └──────────────────────────┬──────────────────────────────────┘
                             │ optional
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │                   PersistentStorage                         │
  │  - Serializes ContentMessage → JSON with hex encoding       │
  │  - Uses HistoryStorage interface (localStorage-compatible)  │
  └─────────────────────────────────────────────────────────────┘

This is not the architecture I originally had in mind, which is fine. It just means that I did some pre-optimisation that now needs to be trashed.

In this proposed architecture, there is only one implementation of ILocalHistory, hence, the interface can be trashed.

MemLocalHistory persist when possible -> so it's a not a Mem history anymore and can just be called LocalHistory.

Then the last part Uses HistoryStorage interface (localStorage-compatible) -> this is confusing. the interface should be called ILocalStorage and not have the history word in it, which is a term use in the SDS domain, and local storage is not SDS-aware.

danisharora099 · 2025-12-09T22:07:01Z

After more review, I now better understand what is done here:

  ┌─────────────────────────────────────────────────────────────┐
  │                     MessageChannel                          │
  │  (defaults to MemLocalHistory with channelId for storage)   │
  └──────────────────────────┬──────────────────────────────────┘
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │                    MemLocalHistory                          │
  │  implements ILocalHistory                                   │
  │  - Manages in-memory message array                          │
  │  - Delegates persistence to PersistentStorage               │
  └──────────────────────────┬──────────────────────────────────┘
                             │ optional
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │                   PersistentStorage                         │
  │  - Serializes ContentMessage → JSON with hex encoding       │
  │  - Uses HistoryStorage interface (localStorage-compatible)  │
  └─────────────────────────────────────────────────────────────┘

Ah, I see where the confusion came from. Thanks for the diagram (note to self to include it wherever possible), I can see how it may have been confusing since the structure changed.

MemLocalHistory persist when possible -> so it's a not a Mem history anymore and can just be called LocalHistory.

Agreed, see my comment here: #2741 (comment)

Then the last part Uses HistoryStorage interface (localStorage-compatible) -> this is confusing. the interface should be called ILocalStorage and not have the history word in it, which is a term use in the SDS domain, and local storage is not SDS-aware.

Valid point, thanks! One caveat: the class is more than just "local storage" as it allows users to pass their custom storage providers, so IStorage, perhaps?

…te constant

…istentHistory` to use composition over inheritance for history management.

…e key generation.

…e`/`load`

…age operations.

…ifying history storage initialization to default to `localStorage`

…mproved compatibility.

…in message channels

…d refactor `MemLocalHistory` to support optional persistent storage.

… persistence, updating LocalHistory to utilize a unified Storage interface and enhancing tests for localStorage functionality

fryorcraken reviewed Nov 15, 2025

View reviewed changes

packages/sds/src/message_channel/persistent_history.ts Outdated Show resolved Hide resolved

fryorcraken reviewed Nov 15, 2025

View reviewed changes

packages/sds/src/message_channel/persistent_history.ts Outdated Show resolved Hide resolved

danisharora099 force-pushed the feat/persistent_history branch from d7aa504 to 92dd0ba Compare November 20, 2025 21:27

danisharora099 requested a review from fryorcraken November 27, 2025 02:28

danisharora099 changed the title ~~feat: persistent history for SDS~~ feat(sds): persistent storage for history Nov 27, 2025

danisharora099 force-pushed the feat/persistent_history branch from df7d56b to 3e3c511 Compare November 27, 2025 03:40

danisharora099 marked this pull request as ready for review November 27, 2025 04:07

danisharora099 requested a review from a team as a code owner November 27, 2025 04:07

fryorcraken reviewed Nov 27, 2025

View reviewed changes

fryorcraken requested changes Nov 29, 2025

View reviewed changes

danisharora099 force-pushed the feat/persistent_history branch from 434728f to 96ec51f Compare December 9, 2025 22:00

danisharora099 added 14 commits December 10, 2025 15:20

feat: persistent history for SDS

dad9cff

refactor: simplify localHistory initialization by removing intermedia…

454dc4a

…te constant

feat(history): introduce ILocalHistory interface and refactor `Pers…

ca24c09

…istentHistory` to use composition over inheritance for history management.

refactor: rename storageKey to storageKeyPrefix and update storag…

032c3f5

…e key generation.

refactor: rename persistence methods from persist/restore to `sav…

b8ee363

…e`/`load`

refactor: Remove silent error suppression for persistent history stor…

322240b

…age operations.

feat: remove storageKeyPrefix and getDefaultHistoryStorage, simpl…

15f1a22

…ifying history storage initialization to default to `localStorage`

feat: Conditionally set default history storage to localStorage for i…

581430a

…mproved compatibility.

tests: use memLocalHistory for message_channel.spec.ts

d52ea9f

test: add unit tests for localStorage persistence and error handling …

d6412e5

…in message channels

test: local storage specific test

ad4973a

feat: Introduce PersistentStorage for message history management an…

503c348

…d refactor `MemLocalHistory` to support optional persistent storage.

chore: remove unused exports

e54dacd

chore: interface abstractions

a3202b5

danisharora099 added 7 commits December 10, 2025 15:20

chore: update interface to type as we're not implementing it

4757322

fix: MemLocalHistory uses optional chaining

ff8b4e5

chore: add logging to LocalHistory for storage init

2ee8b08

chore: address comments, reduce diff

41f8fdf

chore: update word history to storage

721c494

refactor: implement browser and Node.js storage solutions for message…

8a749c4

… persistence, updating LocalHistory to utilize a unified Storage interface and enhancing tests for localStorage functionality

refactor: cleaner API for storage in LocalHistoryOptions

e396c3b

danisharora099 force-pushed the feat/persistent_history branch from 75e8e12 to e396c3b Compare December 10, 2025 20:21

minor fixes

ab6d3cf

danisharora099 force-pushed the feat/persistent_history branch from 8503315 to ab6d3cf Compare December 10, 2025 20:38

danisharora099 requested a review from fryorcraken December 11, 2025 17:09

feat(sds): persistent storage for history #2741

Are you sure you want to change the base?

feat(sds): persistent storage for history #2741

Uh oh!

Conversation

danisharora099 commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem / Description

Solution

Changes

Notes

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danisharora099 Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danisharora099 Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

size-limit report 📦

Uh oh!

danisharora099 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken commented Nov 30, 2025

Uh oh!

danisharora099 commented Nov 14, 2025 •

edited

Loading

danisharora099 Nov 27, 2025 •

edited

Loading

danisharora099 Dec 9, 2025 •

edited

Loading

github-actions bot commented Nov 20, 2025 •

edited

Loading

danisharora099 commented Nov 27, 2025 •

edited

Loading

danisharora099 commented Dec 9, 2025 •

edited

Loading