Skip to content

Commit 03b6a62

Browse files
authored
fix(ocap-kernel): restore IO channels for persisted subclusters (#963)
## Summary - Adds `SubclusterManager.restorePersistedIOChannels()` and calls it from `Kernel.#init` before `initializeAllVats`. - Walks every persisted subcluster, finds those whose config declares `io`, and re-creates the channels via the `IOManager`. - Per-subcluster failures are logged but do not abort the broader init. ## Why Subclusters that declare `config.io` (Unix sockets, named pipes, etc.) had their channels created inside `launchSubcluster` only. On a kernel restart the persisted vats are re-incarnated via `initializeAllVats`, but their IO channels were never re-created — any `IOService` references the vats held went dead at first use after the restart. Concretely: the `@ocap/service-matcher` matcher vat persists across restarts (its OCAP URL is deterministic and its registry is in baggage), and it holds an `IOService` reference for the `llm` socket used to talk to its LLM bridge. After a `daemon stop` / `daemon start` cycle, the matcher vat came back up but the first registration attempt failed inside `ingestService` because the `llm` channel no longer existed in the kernel's `IOManager`. With this fix, the channel is re-established before any vat code runs. ## Test plan - [x] 4 new unit tests in `SubclusterManager.test.ts` covering: multi-subcluster restoration, skipping subclusters with no IO config, no-op when IOManager is absent, and continuing past a per-subcluster failure. - [x] `yarn workspace @MetaMask/ocap-kernel test:dev:quiet --run` — all package tests pass. - [x] `yarn workspace @MetaMask/ocap-kernel lint` — clean. - [ ] CI green. - [ ] End-to-end verification (in a downstream branch): `daemon stop`/`start` cycle now leaves the matcher's `llm` IOService live; first service registration after restart succeeds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches kernel init ordering and external IO resources (sockets); failures are isolated per subcluster but affected vats still break at first IO use until the channel is restored. > > **Overview** > Fixes **dead `IOService` references** after a kernel restart when subclusters were only created once via `launchSubcluster`. > > Adds **`SubclusterManager.restorePersistedIOChannels()`**, which scans persisted subclusters with `config.io` and calls **`IOManager.createChannels`** again (same as at launch). **`Kernel.#init`** now awaits this **after** `initSystemSubclusters` and **before** `initializeAllVats`, so IO kernel services exist before re-incarnated vats run. > > Per-subcluster **`createChannels`** failures are **logged only**; init continues for other subclusters. **No-op** if no `IOManager` was wired. **CHANGELOG** and **four unit tests** cover the new behavior. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 96ed00e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 7c3845b commit 03b6a62

4 files changed

Lines changed: 174 additions & 0 deletions

File tree

packages/ocap-kernel/CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4444
- Use length-prefixed framing for remote messages so payloads larger than the underlying transport's per-frame cutoff (e.g. `@libp2p/webrtc`'s 16 KB datachannel limit) are reassembled correctly on the receiver ([#957](https://github.com/MetaMask/ocap-kernel/pull/957))
4545
- Replace `byteStream` with `lpStream` on every remote channel; the byte-oriented stream did not preserve `write()` boundaries, so any message the transport split into multiple frames was parsed from the first frame only, silently dropped without acknowledgement, and the sender retried until giving up after `MAX_RETRIES`
4646
- Surface receiver-side framing-cap violations (`InvalidDataLengthError`, `InvalidDataLengthLengthError`) as `ResourceLimitError` with `limitType: 'messageSize'` so size errors look the same whether they tripped on the sender's `validateMessageSize` or the receiver's framing decoder
47+
- Restore IO channels for persisted subclusters at kernel init so re-incarnated vats find their IOService references live ([#963](https://github.com/MetaMask/ocap-kernel/pull/963))
48+
- `SubclusterManager.restorePersistedIOChannels()` walks every persisted subcluster, finds those whose config declares `io`, and re-creates the channels via `IOManager` before `initializeAllVats` runs
49+
- Without this, any vat that opened an IO channel via `launchSubcluster` lost its channel across `daemon stop` / `daemon start` and silently held a dead IOService reference
4750

4851
## [0.7.0]
4952

packages/ocap-kernel/src/Kernel.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,12 @@ export class Kernel {
282282
// longer have a config, to ensure that orphaned vats aren't started
283283
this.#subclusterManager.initSystemSubclusters(configs);
284284

285+
// Re-create IO channels for persisted subclusters whose configs
286+
// declare them. This must happen before `initializeAllVats` so
287+
// that re-incarnated vats find their IOService references live
288+
// when they make their first method call.
289+
await this.#subclusterManager.restorePersistedIOChannels();
290+
285291
// Start all vats that were previously running before starting the queue
286292
// This ensures that any messages in the queue have their target vats ready
287293
await this.#vatManager.initializeAllVats();

packages/ocap-kernel/src/vats/SubclusterManager.test.ts

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -821,4 +821,119 @@ describe('SubclusterManager', () => {
821821
).not.toHaveBeenCalled();
822822
});
823823
});
824+
825+
describe('restorePersistedIOChannels', () => {
826+
const makeMockIOManager = () => ({
827+
createChannels: vi.fn().mockResolvedValue(undefined),
828+
destroyChannels: vi.fn().mockResolvedValue(undefined),
829+
});
830+
831+
const subclusterWithIO = (
832+
id: string,
833+
io: Record<string, { type: 'socket'; path: string }>,
834+
): Subcluster => ({
835+
id,
836+
config: {
837+
bootstrap: 'svc',
838+
vats: { svc: { sourceSpec: 'svc.js' } },
839+
io,
840+
},
841+
vats: { svc: 'v1' as VatId },
842+
});
843+
844+
const subclusterWithoutIO = (id: string): Subcluster => ({
845+
id,
846+
config: createMockClusterConfig(`s-${id}`),
847+
vats: { [`s-${id}Vat`]: 'v1' as VatId },
848+
});
849+
850+
it('re-creates channels for every persisted subcluster declaring io', async () => {
851+
const ioA = { llm: { type: 'socket' as const, path: '/tmp/a.sock' } };
852+
const ioB = { wire: { type: 'socket' as const, path: '/tmp/b.sock' } };
853+
mockKernelStore.getSubclusters.mockReturnValue([
854+
subclusterWithIO('s1', ioA),
855+
subclusterWithIO('s2', ioB),
856+
]);
857+
const ioManager = makeMockIOManager();
858+
const mgr = new SubclusterManager({
859+
kernelStore: mockKernelStore,
860+
kernelQueue: mockKernelQueue,
861+
vatManager: mockVatManager,
862+
getKernelService: mockGetKernelService,
863+
queueMessage: mockQueueMessage,
864+
ioManager: ioManager as never,
865+
});
866+
867+
await mgr.restorePersistedIOChannels();
868+
869+
expect(ioManager.createChannels).toHaveBeenCalledTimes(2);
870+
expect(ioManager.createChannels).toHaveBeenCalledWith('s1', ioA);
871+
expect(ioManager.createChannels).toHaveBeenCalledWith('s2', ioB);
872+
});
873+
874+
it('skips subclusters whose config does not declare io', async () => {
875+
mockKernelStore.getSubclusters.mockReturnValue([
876+
subclusterWithoutIO('s1'),
877+
subclusterWithIO('s2', {
878+
io: { type: 'socket' as const, path: '/tmp/s2.sock' },
879+
}),
880+
]);
881+
const ioManager = makeMockIOManager();
882+
const mgr = new SubclusterManager({
883+
kernelStore: mockKernelStore,
884+
kernelQueue: mockKernelQueue,
885+
vatManager: mockVatManager,
886+
getKernelService: mockGetKernelService,
887+
queueMessage: mockQueueMessage,
888+
ioManager: ioManager as never,
889+
});
890+
891+
await mgr.restorePersistedIOChannels();
892+
893+
expect(ioManager.createChannels).toHaveBeenCalledTimes(1);
894+
expect(ioManager.createChannels).toHaveBeenCalledWith(
895+
's2',
896+
expect.objectContaining({ io: expect.anything() }),
897+
);
898+
});
899+
900+
it('is a no-op when no IOManager was provided', async () => {
901+
mockKernelStore.getSubclusters.mockReturnValue([
902+
subclusterWithIO('s1', {
903+
llm: { type: 'socket' as const, path: '/tmp/a.sock' },
904+
}),
905+
]);
906+
// Default subclusterManager from beforeEach has no ioManager;
907+
// a missing manager must not cause an error here.
908+
expect(
909+
await subclusterManager.restorePersistedIOChannels(),
910+
).toBeUndefined();
911+
});
912+
913+
it('continues past a per-subcluster failure', async () => {
914+
mockKernelStore.getSubclusters.mockReturnValue([
915+
subclusterWithIO('s1', {
916+
llm: { type: 'socket' as const, path: '/tmp/a.sock' },
917+
}),
918+
subclusterWithIO('s2', {
919+
wire: { type: 'socket' as const, path: '/tmp/b.sock' },
920+
}),
921+
]);
922+
const ioManager = makeMockIOManager();
923+
ioManager.createChannels.mockImplementationOnce(async () => {
924+
throw new Error('socket in use');
925+
});
926+
const mgr = new SubclusterManager({
927+
kernelStore: mockKernelStore,
928+
kernelQueue: mockKernelQueue,
929+
vatManager: mockVatManager,
930+
getKernelService: mockGetKernelService,
931+
queueMessage: mockQueueMessage,
932+
ioManager: ioManager as never,
933+
});
934+
935+
expect(await mgr.restorePersistedIOChannels()).toBeUndefined();
936+
expect(ioManager.createChannels).toHaveBeenCalledTimes(2);
937+
});
938+
});
824939
});

packages/ocap-kernel/src/vats/SubclusterManager.ts

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,56 @@ export class SubclusterManager {
368368
this.#restorePersistedSystemSubclusters(configs);
369369
}
370370

371+
/**
372+
* Re-create IO channels for every persisted subcluster whose config
373+
* declares one.
374+
*
375+
* IO channels (`config.io`) are external to the kernel's durable
376+
* state — they're live OS-process resources (Unix sockets, pipes,
377+
* etc.) backed by the host's IOManager. `launchSubcluster` creates
378+
* them at subcluster-launch time, but on a kernel restart the vats
379+
* get re-incarnated from persisted state without going through
380+
* `launchSubcluster` again, so any IO channels they were using are
381+
* gone. This method bridges that gap: invoked from the kernel's
382+
* init path before `initializeAllVats`, it walks the persisted
383+
* subcluster table and re-creates each subcluster's declared IO
384+
* channels so the re-incarnated vats find their IOService
385+
* references live and ready when they make their first method call.
386+
*
387+
* If a particular subcluster's `createChannels` call fails (e.g.
388+
* the socket path is now in use by another process), the failure
389+
* is logged but does not abort the broader init — other
390+
* subclusters can still come up. Vats backed by the failed
391+
* subcluster will see method calls on their IOServices fail at
392+
* first use.
393+
*
394+
* No-op when no IOManager was provided to this SubclusterManager.
395+
*/
396+
async restorePersistedIOChannels(): Promise<void> {
397+
if (!this.#ioManager) {
398+
return;
399+
}
400+
for (const subcluster of this.#kernelStore.getSubclusters()) {
401+
if (!subcluster.config.io) {
402+
continue;
403+
}
404+
try {
405+
await this.#ioManager.createChannels(
406+
subcluster.id,
407+
subcluster.config.io,
408+
);
409+
this.#logger.info(
410+
`Restored IO channels for persisted subcluster ${subcluster.id}`,
411+
);
412+
} catch (error) {
413+
this.#logger.error(
414+
`Failed to restore IO channels for subcluster ${subcluster.id}:`,
415+
error,
416+
);
417+
}
418+
}
419+
}
420+
371421
/**
372422
* Launch new system subclusters that aren't already in persistence.
373423
* This must be called after the kernel queue is running since launchSubcluster

0 commit comments

Comments
 (0)