Skip to content

Commit 34e8ec2

Browse files
rekmarksclaude
andcommitted
feat(netlayer): extract @metamask/netlayer + @metamask/netlayer-loopback (netlayer phase 3)
Extract the transport-neutral networking machinery and the netlayer contract out of ocap-kernel's remotes/platform into a new published package @metamask/netlayer (types, channel-session engine `makeChannelNetlayer`, shared machinery, neutral identity helpers), and add @metamask/netlayer-loopback, an in-process hub netlayer used as the standard test fake. - @metamask/netlayer: moves peer-state-manager, reconnection(+lifecycle), rate-limiter, validators, channel-utils, handshake (now versioned), the neutral engine constants, identity helpers, and the engine (transport.ts -> channel-netlayer.ts) with their tests. ChannelProvider/Netlayer gain a readonly peerId. - @metamask/netlayer-loopback: makeLoopbackHub + makeLoopbackNetlayer routing messages between Netlayers in one realm, keyed by neutral peerId. - ocap-kernel keeps the libp2p ConnectionFactory (now a ChannelProvider with a peerId getter) and a thin initTransport that delegates to makeChannelNetlayer; remotes/types.ts re-exports the netlayer contract types; connection-factory and remote-comms import identity/constants from @metamask/netlayer. - Demonstration: loopback-backed PlatformServices test util + a two-party in-process message/reply test through initializeRemoteComms. Runtimes are untouched (initTransport signature unchanged). Part of the netlayer plan (issue #968), stacked on rekm/netlayer-2 (#971). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 791db3e commit 34e8ec2

64 files changed

Lines changed: 6303 additions & 4096 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/plans/netlayer/master.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -254,6 +254,98 @@ Move shared machinery + engine + types (with tests) out of `remotes/platform/`;
254254
re-exports types. Implement loopback (in-memory hub keyed by peerId); use it to replace
255255
hand-rolled `PlatformServices` mocks in some ocap-kernel tests. ocap-kernel still hosts the
256256
libp2p provider, so runtimes are untouched. Verify: full suite, `yarn constraints`.
257+
**DONE** (issue #968, PR on branch `rekm/netlayer-3`, stacked on `rekm/netlayer-2`).
258+
259+
Landed decisions / deviations from the phase-3 plan (these are the facts Phase 4 must build on):
260+
261+
- **`@metamask/netlayer` (new, `0.1.0`, published, single `.` export).** Barrel exports:
262+
- Engine: `makeChannelNetlayer({ provider, hooks, options, logger, stopController }) => Netlayer`
263+
(synchronous; **kept the Phase-1 landed signature**`stopController` is passed through, not a
264+
bare `incarnationId` param; the local incarnation lives in `options.localIncarnationId`).
265+
- Identity: `deriveNeutralPeerId`, `neutralPeerIdToPublicKey`, `publicKeyToNeutralPeerId` (moved
266+
verbatim from `remotes/kernel/identity.ts`).
267+
- Shared machinery: `PeerStateManager`, `ReconnectionManager`, `PERMANENT_FAILURE_ERROR_CODES`,
268+
`makeReconnectionLifecycle`, `makeMessageRateLimiter`, `makeConnectionRateLimiter`,
269+
`SlidingWindowRateLimiter`, `makeMessageSizeValidator`, `makeConnectionLimitChecker`,
270+
`makeErrorLogger`, `writeWithTimeout`, `performInboundHandshake`, `performOutboundHandshake`,
271+
`isHandshakeMessage`, `HANDSHAKE_VERSION` (= `1`), plus the neutral engine constants
272+
(`DEFAULT_MAX_CONCURRENT_CONNECTIONS`, `DEFAULT_MAX_MESSAGE_SIZE_BYTES`,
273+
`DEFAULT_CLEANUP_INTERVAL_MS`, `DEFAULT_STALE_PEER_TIMEOUT_MS`, `DEFAULT_WRITE_TIMEOUT_MS`,
274+
`DEFAULT_MESSAGE_RATE_LIMIT`, `DEFAULT_MESSAGE_RATE_WINDOW_MS`, `DEFAULT_CONNECTION_RATE_LIMIT`,
275+
`DEFAULT_CONNECTION_RATE_WINDOW_MS`, `HANDSHAKE_TIMEOUT_MS`, `STREAM_INACTIVITY_TIMEOUT_MS`,
276+
`MIN_STREAM_INACTIVITY_TIMEOUT_MS`, `DEFAULT_CONSECUTIVE_ERROR_THRESHOLD`).
277+
- Types: `Netlayer`, `NetlayerHooks`, `NetlayerParams`, `NetlayerFactory`, `NetlayerSpecifier`,
278+
`NetlayerRegistry`, `NetworkChannel`, `ChannelProvider`, `InboundChannelHandler`,
279+
`PeerDisconnectHandler`, `RemoteMessageHandler`, `SendRemoteMessage`, `StopRemoteComms`,
280+
`OnRemoteGiveUp`, `OnIncarnationChange`, `ChannelNetlayerOptions`, `PeerState`,
281+
`ReconnectionState`, `ErrorRecord`, `HandshakeMessage`, `HandshakeResult`, `HandshakeDeps`,
282+
`ErrorLogger`, `ReconnectionLifecycle`, `ReconnectionLifecycleDeps`.
283+
- Deps: `@metamask/kernel-errors`, `@metamask/kernel-utils`, `@metamask/logger`,
284+
`@metamask/utils` (`Json`), `@noble/curves`, `multiformats`, `uint8arrays`. **`@metamask/superstruct`
285+
was dropped** (the plan listed it, but netlayer defines no struct helpers yet — depcheck flags
286+
unused deps; add it back in Phase 5 when the websocket netlayer needs config structs).
287+
`@types/node` is a devDep. No libp2p anywhere in its tree.
288+
- **`ChannelProvider` gained `readonly peerId: string`** and `Netlayer` gained `readonly peerId`
289+
(Phase-3 work per plan §3). `ConnectionFactory` implements it via a `get peerId()` returning a
290+
`#neutralPeerId` field set in the constructor as `deriveNeutralPeerId(fromHex(options.keySeed))`.
291+
The engine's returned `Netlayer.peerId` is `provider.peerId`. The returned netlayer is `harden`ed.
292+
- **Handshake is versioned.** `HandshakeMessage` now carries `v: number` (current `HANDSHAKE_VERSION = 1`),
293+
sent on every handshake/handshakeAck; `isHandshakeMessage` requires a numeric `v`. Compat is broken
294+
by design.
295+
- **What moved to `@metamask/netlayer/src/`** (tests moved with each): `peer-state-manager.ts`,
296+
`reconnection.ts`, `reconnection-lifecycle.ts`, `rate-limiter.ts`, `validators.ts`,
297+
`channel-utils.ts`, `handshake.ts`, `identity.ts`, the neutral subset of `constants.ts`, and
298+
`transport.ts`'s `makeChannelNetlayer` engine → `channel-netlayer.ts` (+ its test as
299+
`channel-netlayer.test.ts`, which now drives the engine through a mock `ChannelProvider` via a
300+
local `initTransport` shim; the three `ConnectionFactory.make`-argument assertions stayed in
301+
ocap-kernel). New files: `types.ts`, `index.ts`.
302+
- **What ocap-kernel keeps in `remotes/platform/`:** `connection-factory.ts` (+test) — still the
303+
libp2p `ChannelProvider`, now importing `NetworkChannel`/`InboundChannelHandler`/
304+
`PeerDisconnectHandler` + `DEFAULT_MAX_MESSAGE_SIZE_BYTES` + `deriveNeutralPeerId`/
305+
`neutralPeerIdToPublicKey`/`publicKeyToNeutralPeerId` from `@metamask/netlayer` (its
306+
`connection-factory.test.ts` now `vi.mock('@metamask/netlayer', ...)` with `importOriginal` to
307+
stub the three identity helpers); `lp-framing.test.ts`; a reduced `constants.ts` (libp2p-only:
308+
`SCTP_USER_INITIATED_ABORT`, `RELAY_RECONNECT_*`); and a **new thin `transport.ts`** whose
309+
`initTransport(...)` (unchanged public signature, now returns `Netlayer`) builds `ConnectionFactory`
310+
and delegates to `makeChannelNetlayer`. `remote-comms.ts` imports `deriveNeutralPeerId` from
311+
`@metamask/netlayer` (its ocap-URL/AES-GCM code stays put).
312+
- **ocap-kernel re-export surface:** `remotes/types.ts` re-exports the netlayer contract types
313+
(`NetworkChannel`, `ChannelProvider`, `RemoteMessageHandler`, `SendRemoteMessage`, `StopRemoteComms`,
314+
`OnRemoteGiveUp`, `OnIncarnationChange`, `Netlayer`, `NetlayerHooks`, `NetlayerFactory`,
315+
`NetlayerParams`, `NetlayerSpecifier`, `NetlayerRegistry`, `InboundChannelHandler`,
316+
`PeerDisconnectHandler`) and keeps `RemoteIdentity`/`RemoteComms`/`RemoteInfo`/`RemoteCommsOptions`/
317+
`DirectTransport`/`ConnectionFactoryOptions` local. `src/index.ts` still exports `initTransport` and
318+
the three identity helpers (now sourced from `@metamask/netlayer`), and adds the netlayer type
319+
surface. `PlatformServices` shape is unchanged. Deps: `@metamask/netlayer` (dependency),
320+
`@metamask/netlayer-loopback` (devDependency, test fake).
321+
- **`@metamask/netlayer-loopback` (new, `0.1.0`, published, single `.` export).** Exports
322+
`makeLoopbackHub`/`LoopbackHub`/`LoopbackRegistration` and `makeLoopbackNetlayer`/`LoopbackConfig`.
323+
The hub is an **explicit object passed via `config.hub`** (no global state); `register(peerId,
324+
receive, incarnationId)` / `unregister` / `getIncarnation` / `deliver(from, to, msg) => reply`.
325+
`makeLoopbackNetlayer` derives its peerId with `deriveNeutralPeerId(fromHex(keySeed))`, registers
326+
with the hub, and returns a hardened `Netlayer`: `sendRemoteMessage` routes through
327+
`hub.deliver` and feeds any reply back into its own `handleMessage` (mirroring the engine's reply
328+
path); `onIncarnationChange` fires once per peer on first contact with the peer's registered
329+
incarnation (no handshake); `closeConnection` marks a peer so sends throw `IntentionalCloseError`
330+
(cleared by `reconnectPeer`); `stop` unregisters and makes sends throw `NetworkStoppedError`;
331+
`registerLocationHints`/`resetAllBackoffs` are no-ops; `getListenAddresses` returns `[]`.
332+
Config `hub` is a live object (not `Json`), so a loopback specifier can't cross postMessage —
333+
same-realm only. Deps: `@metamask/kernel-errors` (**added — the plan's §2.2 list omitted it but
334+
the throw-`NetworkStoppedError`/`IntentionalCloseError` behavior needs it**), `@metamask/kernel-utils`,
335+
`@metamask/netlayer`, `@metamask/superstruct`; `@metamask/logger` was **not** added (unused).
336+
- **Demonstration:** `packages/ocap-kernel/test/loopback-platform-services.ts` provides
337+
`makeLoopbackPlatformServices({ hub })` (its per-call `keySeed`/`incarnationId` come from
338+
`initializeRemoteComms`'s arguments — the plan's construction-time `keySeed`/`incarnationId` were
339+
dropped as redundant), and `packages/ocap-kernel/src/remotes/loopback.test.ts` is a two-party
340+
in-process message+reply test through the real `PlatformServices.initializeRemoteComms` path.
341+
**`RemoteManager.test.ts` was NOT converted** to loopback (the plan's optional §6.3) — the new
342+
two-party test is the credible demonstration; `makeMockPlatformServices` stays as-is.
343+
- **Coverage:** `@metamask/netlayer` 90.45% stmts / 90.31% branch / 88.46% func / 90.39% line;
344+
`@metamask/netlayer-loopback` 97.56% stmts / 95.83% branch. `@metamask/ocap-kernel` statements,
345+
functions, and lines all rose above the pre-phase baseline (93.24→93.7%, 93.85→94.4%, 93.19→93.7%);
346+
**branch coverage fell ~0.4% (85.9→85.5%)** — a purely arithmetic effect of extracting ~247 branches
347+
that were covered at ~91.5% (well above the package average) into netlayer, where they remain
348+
covered at 90.31%. No source lost test coverage.
257349

258350
**Phase 4 — Extract `@metamask/netlayer-libp2p` + runtime injection.** ~5–6 days (split 4a/4b).
259351

docs/plans/netlayer/phase-4.md

Lines changed: 75 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -9,51 +9,69 @@ leave CI green.
99

1010
## Assumed starting state (Phases 1–3 done)
1111

12-
> **Revision required before execution.** Reconcile every cross-phase reference (names,
13-
> signatures, file locations) against the actually-merged Phases 1–3; where they differ,
14-
> the landed code wins and this document should be updated first.
15-
16-
This plan is written against the current `main`, but Phases 1–3 land first. Where the
17-
current code differs from the assumed post-Phase-3 state, the difference is called out
18-
as a **DELTA**. The assumed state:
19-
20-
- **`@metamask/netlayer` exists** and owns the kernel-facing contract types
21-
(`Netlayer`, `NetlayerFactory`, `NetlayerParams`, `NetlayerHooks`, `NetlayerSpecifier`,
22-
`NetlayerRegistry`), the channel seam (`NetworkChannel`, `ChannelProvider`), the
23-
channel-session engine `makeChannelNetlayer` (refactored `transport.ts`), the shared
24-
machinery (`handshake.ts`, `peer-state-manager.ts`, `reconnection.ts`,
25-
`reconnection-lifecycle.ts`, `rate-limiter.ts`, `validators.ts`, `channel-utils.ts`,
26-
the networking `constants.ts`), and neutral Ed25519 identity helpers. Their tests moved
27-
with them.
28-
- DELTA: today these live in `packages/ocap-kernel/src/remotes/platform/` and
29-
`transport.ts` is a monolithic `initTransport`, not `makeChannelNetlayer`.
30-
- **`@metamask/netlayer-loopback` exists** (in-memory hub `ChannelProvider`/`Netlayer`),
31-
used to replace some hand-rolled `PlatformServices` mocks in ocap-kernel tests.
32-
- **Identity is neutral.** `remotes/kernel/remote-comms.ts` derives the peerId via
33-
`@noble/curves` + WebCrypto; `@libp2p/crypto`/`@libp2p/peer-id` are gone from the kernel
34-
identity path; the `ocap:` URL host is the neutral id; hints are opaque strings.
35-
- DELTA: today `remote-comms.ts` uses `@libp2p/crypto` + `@libp2p/peer-id`, and
36-
`kernel-test/src/remote-comms.test.ts` still imports them to derive the peerId.
37-
- **kernel-errors carries neutral error classes** (`ChannelResetError`,
38-
`IntentionalDisconnectError`, `MessageTooLargeError`) and no longer imports libp2p; the
39-
libp2p→neutral error mapper lives in `remotes/platform/` (created in Phase 1) as the
40-
only remaining libp2p-aware error code.
41-
- DELTA: today error classification is inline in `transport.ts`
42-
(`@libp2p/interface`, `@libp2p/utils` imports) and in `reconnection.ts`
43-
(`PERMANENT_FAILURE_ERROR_CODES`), and `isRetryableNetworkError` in kernel-errors
44-
still name-sniffs libp2p `MuxerClosedError`.
45-
- **ocap-kernel still physically hosts the libp2p implementation**: a libp2p
46-
`ChannelProvider` (`remotes/platform/connection-factory.ts`), the libp2p error mapper,
47-
`utils/multiaddr.ts`, `utils/network.ts`, and a thin `initTransport` that constructs the
48-
provider and hands it to `makeChannelNetlayer`. It re-exports the netlayer types from
49-
`@metamask/netlayer`. It still carries all libp2p deps.
50-
- DELTA: today `connection-factory.ts` produces the old libp2p `Channel`
51-
(`{ msgStream, stream, peerId }`), not `NetworkChannel`, and `initTransport` is the
52-
monolith in `transport.ts`.
53-
54-
Because of these deltas, the file-move maps below name the assumed post-Phase-3 files
55-
(e.g. "connection-factory.ts, now a `ChannelProvider`"). If Phase 3 slipped, do that
56-
extraction first — do not fold it into Phase 4.
12+
> **Reconciled against landed Phases 1–3** (branches `rekm/netlayer-1/2/3`). The prior
13+
> DELTA notes are resolved; the state below is what actually merged. Read master.md's
14+
> "Phase 3 — Landed decisions" block for the exact `@metamask/netlayer` /
15+
> `@metamask/netlayer-loopback` surface. Cross-phase line numbers elsewhere are indicative;
16+
> the landed code wins.
17+
18+
Landed and true as of the start of Phase 4:
19+
20+
- **`@metamask/netlayer` exists** (published, `0.1.0`, single `.` export) and owns the
21+
kernel-facing contract types (`Netlayer`, `NetlayerFactory`, `NetlayerParams`,
22+
`NetlayerHooks`, `NetlayerSpecifier`, `NetlayerRegistry`), the channel seam
23+
(`NetworkChannel`, `ChannelProvider` — the latter now carries `readonly peerId`), the
24+
channel-session engine `makeChannelNetlayer` (`src/channel-netlayer.ts`), the shared
25+
machinery (`handshake.ts` — now with a versioned message, `peer-state-manager.ts`,
26+
`reconnection.ts`, `reconnection-lifecycle.ts`, `rate-limiter.ts`, `validators.ts`,
27+
`channel-utils.ts`, the neutral engine `constants.ts`), and the neutral Ed25519 identity
28+
helpers (`identity.ts`: `deriveNeutralPeerId`/`neutralPeerIdToPublicKey`/
29+
`publicKeyToNeutralPeerId`). Tests moved with them.
30+
- **`makeChannelNetlayer` signature (LANDED, use this — the §2.3/§2.6 sketches below
31+
predate it):** `makeChannelNetlayer({ provider, hooks, options, logger, stopController })`
32+
returning `Netlayer` (synchronous). It does **not** take a bare `incarnationId`; the
33+
local incarnation is `options.localIncarnationId`. `stopController` is a shared
34+
`AbortController``stop()` aborts it and it is the same signal the provider was
35+
constructed with. `make-libp2p-netlayer.ts` must construct that `AbortController`, pass
36+
`signal` to `ConnectionFactory.make` and the whole controller to `makeChannelNetlayer`
37+
(this is exactly what ocap-kernel's thin `initTransport` does today — copy it).
38+
- **`@metamask/netlayer-loopback` exists** (published, `0.1.0`). It is **not** a
39+
`ChannelProvider` — it implements `Netlayer` directly over an in-process `LoopbackHub`
40+
(`makeLoopbackHub`/`makeLoopbackNetlayer`, config `{ hub }`). Used as the standard test
41+
fake; `packages/ocap-kernel/test/loopback-platform-services.ts` wraps it as a
42+
`PlatformServices` and `packages/ocap-kernel/src/remotes/loopback.test.ts` is a two-party
43+
in-process demonstration.
44+
- **Identity is neutral (Phase 2, unchanged in 3).** `remotes/kernel/remote-comms.ts`
45+
derives the peerId via `deriveNeutralPeerId` (now imported from `@metamask/netlayer`) +
46+
WebCrypto for the ocap-URL oid; `@libp2p/crypto`/`@libp2p/peer-id` are gone from the
47+
kernel identity path; the `ocap:` URL host is the neutral id; hints are opaque strings.
48+
- **Still-open DELTA for Phase 4:** `kernel-test/src/remote-comms.test.ts`'s fake
49+
platform-services derives its peerId with `deriveNeutralPeerId` (from `@metamask/ocap-kernel`),
50+
**not** `@libp2p/*` — that was already fixed in Phase 2. Its `initializeRemoteComms` is
51+
still the positional signature; update it to the 4b options-bag shape.
52+
- **kernel-errors is libp2p-free (Phase 2).** It exports the neutral error classes
53+
(`ChannelResetError`, `IntentionalDisconnectError`, `MessageTooLargeError`); the
54+
libp2p→neutral read-error mapper is `mapLibp2pReadError`/`isIntentionalDisconnect`,
55+
**module-private functions in `connection-factory.ts`** (not a standalone file yet).
56+
`isRetryableNetworkError` still name-sniffs `MuxerClosedError`/`Dial`/`Transport`/
57+
`NO_RESERVATION` **by string** (annotated in kernel-errors to move to netlayer-libp2p's
58+
error mapper here in Phase 4).
59+
- **ocap-kernel still physically hosts the libp2p implementation:** the libp2p
60+
`ChannelProvider` (`remotes/platform/connection-factory.ts`, which now has a
61+
`get peerId()` returning `deriveNeutralPeerId(fromHex(keySeed))` and imports
62+
`NetworkChannel`/`InboundChannelHandler`/`PeerDisconnectHandler` +
63+
`DEFAULT_MAX_MESSAGE_SIZE_BYTES` + the three identity helpers from `@metamask/netlayer`),
64+
the module-private error mapper, `utils/multiaddr.ts`, `utils/network.ts`, a reduced
65+
`constants.ts` (**libp2p-only: `SCTP_USER_INITIATED_ABORT`, `RELAY_RECONNECT_BASE_DELAY_MS`,
66+
`RELAY_RECONNECT_MAX_DELAY_MS`, `RELAY_RECONNECT_MAX_ATTEMPTS`** — the engine constants
67+
moved to `@metamask/netlayer` in Phase 3), and a thin `transport.ts` whose
68+
`initTransport` constructs `ConnectionFactory` and delegates to `makeChannelNetlayer`. It
69+
re-exports the netlayer types and depends on `@metamask/netlayer`. It still carries all
70+
libp2p deps.
71+
72+
The file-move maps below assume this landed state. Where a sketch (esp. §2.3/§2.6) shows
73+
`makeChannelNetlayer({ ..., incarnationId, ...engineOptions })`, use the landed signature
74+
above instead.
5775

5876
---
5977

@@ -186,8 +204,18 @@ Notes:
186204
importer is `connection-factory.ts`.
187205
- `constants.ts` is shared: the timeout/limit constants consumed by `makeChannelNetlayer`
188206
already moved to `@metamask/netlayer` in Phase 3; only the libp2p-provider-only
189-
constants (relay reconnect backoff, SCTP abort code) come to netlayer-libp2p. Import the
190-
shared ones from `@metamask/netlayer` where the provider needs them.
207+
constants (`RELAY_RECONNECT_BASE_DELAY_MS`, `RELAY_RECONNECT_MAX_DELAY_MS`,
208+
`RELAY_RECONNECT_MAX_ATTEMPTS`, `SCTP_USER_INITIATED_ABORT`) come to netlayer-libp2p.
209+
`DEFAULT_MAX_MESSAGE_SIZE_BYTES` stays in `@metamask/netlayer``connection-factory.ts`
210+
already imports it (and the three identity helpers) from there, so those imports need no
211+
change when the file moves (netlayer-libp2p depends on netlayer).
212+
- The "libp2p error mapper (from Phase 1)" is not a standalone file today: it is the
213+
module-private `mapLibp2pReadError` + `isIntentionalDisconnect` functions at the top of
214+
`connection-factory.ts` (covered by `connection-factory.test.ts`, which stubs the
215+
identity helpers via `vi.mock('@metamask/netlayer', ...)`). Extracting them to
216+
`error-mapper.ts` (+ a new test) is Phase-4 work; also fold in the
217+
`MuxerClosedError`/`Dial`/`Transport`/`NO_RESERVATION` name-sniffing that still lives in
218+
kernel-errors' `isRetryableNetworkError`.
191219

192220
### 2.3 Factory signatures
193221

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [Unreleased]
9+
10+
[Unreleased]: https://github.com/MetaMask/ocap-kernel/

0 commit comments

Comments
 (0)