Large neuron meshes never render: OBJ is streamed through the websocket instead of loaded by the client

[Robbie1977]

Summary

Neurons with large arbors never render in the 3D viewer. The mesh is fetched by the server, embedded in the Geppetto model, and pushed to the browser as a single uncompressed websocket text frame. For APL that frame is over 500 MB, the blocking send times out, and the connection is left unusable.

Example: https://v2.virtualflybrain.org/org.geppetto.frontend/geppetto?id=VFB_jrmc3hpm&i=VFB_00101567 (APL_L, MaleCNS v0.9). Term info loads; no geometry ever appears and no error reaches the user.

What the log shows

ERROR DefaultMessageSender  Error sending text message java.net.SocketTimeoutException
WARN  DefaultMessageSender  Failed to send binary message java.lang.IllegalStateException:
      The remote endpoint was in state [TEXT_FULL_WRITING] which is an invalid state for called method
  at org.geppetto.frontend.messaging.DefaultMessageSender.sendTextMessage(DefaultMessageSender.java:369)
  at org.geppetto.frontend.controllers.ConnectionHandler.resolveImportType(ConnectionHandler.java:524)

Tomcat's blocking send timeout fires part-way through the write, leaving the endpoint in TEXT_FULL_WRITING. Every later message on that connection then fails, which is where the trailing broken-pipe, Connection reset by peer and A project without a runtime project cannot be closed entries come from. Those are consequences, not causes, as are the other.getUser() returned null and getAllGeppettoProjects NPE lines, which appear constantly on healthy sessions.

Mesh sizes

Measured against the data host on 2026-08-25, all aligned to JRC2018U (VFB_00101567) unless stated:

Neuron	volume_man.obj	volume.obj	volume.swc
APL_L MaleCNS VFB_jrmc3hpm	513,692,448	404	3,386,579
APL_R MaleCNS VFB_jrmc3hpl	539,778,360	404	3,287,206
APL_R OpticLobe VFB_00103m9x	656,186,044	3,526,734	2,195,159
APL FlyWire VFB_fw036977	295,091,332	295,091,332	10,770,191
APL_R hemibrain VFB_jrchjrhd on JRCFIB2018Fum	absent	7,537,079	8,182,746
10 sampled MaleCNS Kenyon cells	3.7–6.3 MB	404	~70 kB

The APL_L file is trimesh ASCII with roughly 5.68 M vertices and 11 M triangles, undecimated. nginx serves .obj without gzip. The problem scales with arbor size rather than with dataset: hemibrain manages the same cell in 7.5 MB, so a 656 MB OpticLobe APL is the same defect hidden behind a working volume.obj.

Why the SWC fallback does not save it

components/VFBMain.js:587 prefers the full mesh — if the _obj import type's URL contains volume_man.obj it selects that branch and never reaches _swc. The KB returns volume_man.obj in the obj field for every JRC2018U-aligned image across jrmc, 0010, fw03 and jrch; only hemibrain's own template VFB_00101384 returns volume.obj. MaleCNS has no volume.obj at all, so there is nothing smaller to fall back to.

The mesh should not travel through the websocket

ThreeDEngine.loadThreeOBJModelFromNode calls loader.parse(node.obj) — the OBJ text arrives as a property on the model node, which is why the server has to read the file, hold it in heap, serialise it into the model and stream it over the socket. The client already has everything needed to do this itself:

• THREE.OBJLoader.load(url, onLoad, onProgress, onError) exists in the bundled loader (geppetto-client/js/components/interface/3dCanvas/OBJLoader.js:42) and is unused on this path.
• The URL is already on the unresolved import type client-side — VFBMain.js:587 reads getUrl() before any resolve happens.
• The data host already allows it: access-control-allow-origin: *, accept-ranges: bytes, cache-control: max-age=31536000, public, immutable.

Loading the mesh directly in the browser removes the server round-trip and the heap copy, lets the browser cache the file for a year, and gives real load progress instead of a silent stall. It also makes gzip and range requests available, neither of which the websocket path can use.

Two things to fix while doing it: the model should carry the URL rather than the geometry, and the response currently ends with a stray cache-control: private that contradicts the earlier public, immutable and will defeat shared caching.

Proposed work

1. Pass the OBJ URL to the client and load it with THREE.OBJLoader.load, keeping colour, opacity and visibility wired to the existing instance handle rather than bypassing the model.
2. Decimate on publication. Client-side loading fixes transport and the server, but 11 M triangles will still exhaust a laptop browser — a 5.68 M-vertex mesh is roughly 140 MB of GPU buffers before the parse overhead. Target the ~10 MB that hemibrain already achieves.
3. Generate volume.obj for the MaleCNS release, and prefer it over volume_man.obj in the KB obj field where it exists.
4. Add a size guard so a pathological mesh degrades to SWC instead of hanging. The OBJ 503 retry added in org.geppetto.core VFBv2.3.8.1 already returns an empty VisualType and lets the rest of the model load — the same shape works here.
5. Drop the duplicate cache-control: private header on the data host, and enable gzip for .obj.

Items 2 and 3 are the pipeline side and can proceed independently of 1 and 4.

[Robbie1977]

Two corrections and one new finding.

Correction: heap is not the constraint. An earlier reading of ENV MAXSIZE=2G in the Dockerfile was wrong — the deployed service overrides it to 45G. There is no heap pressure and no OOM risk in this path.

Correction: there is no system property for the send timeout. From the Tomcat 8.5 source, WsRemoteEndpointImplBase.getBlockingSendTimeout() reads only the per-session user properties, and Constants.DEFAULT_BLOCKING_SEND_TIMEOUT is a compile-time constant of 20 000 ms. Raising it means setting org.apache.tomcat.websocket.BLOCKING_SEND_TIMEOUT as a Long in WebsocketConnection.onOpen; there is no JAVA_OPTS route.

New finding: the transport has a hard ceiling of roughly 503 MB, independent of any timeout. GeppettoTransportMessage.data is a String, so the model JSON is escaped twice — once when the OBJ text is embedded in the model, again when Gson serialises data. Measured against the APL_L file:

raw bytes                 513,692,448
newlines                   17,079,378   (no quotes or backslashes)
after 1x JSON escape      530,771,826 chars
after 2x (Gson data)      547,851,204 chars
V8 MAX_STRING_LENGTH      536,870,888 chars
over by                    10,980,316 chars  (2.05%)

V8's maximum string length is 536,870,888 characters, so this message cannot exist in the browser as a string. Compression does not change that: the client inflates with pako.ungzip(..., { to: "string" }), back into one JS string. Mesh counts for the record: 5,677,124 vertices, 11,402,252 faces.

That splits the affected neurons into two groups. FlyWire APL (295 MB) is under the ceiling and fails only on time. The MaleCNS APLs (514 and 540 MB) and the OpticLobe APL (656 MB) are above it and cannot be delivered this way at any timeout, which makes the size guard in item 4 a limit the transport genuinely has rather than a convenience.

One more detail on item 4. queuedMessageTypes is commented out in app-config.xml, so isQueuedMessageType() is false for every type and preprocessAndSendMessage runs on the calling thread — the http-nio-8080-exec-N connector thread, as the stack trace shows. queuingEnabled=true has no effect. A long send timeout therefore holds a connector thread for its full duration, which argues for a modest timeout plus a size cap rather than a large timeout alone.

Landed on development for testing: websocket compression enabled (app-config.xml), and an SWC fallback that HEADs the mesh and drops to the skeleton above maxObjBytes, default 150 MB. The ENV-driven send timeout follows separately.

[Robbie1977]

Fixed by #1714 — containment in the viewer and the transport, verified on v2-dev.

APL_L now renders from its skeleton instead of stalling, and meshes in the 83–126 MB range that previously exceeded the 20 s send timeout deliver in under 4 s. Compression is on, the size guard falls back to SWC, the visibility controls no longer bypass it, and a connection whose send has failed is now closed so the client reconnects rather than the session dying silently.

The underlying data problem is unchanged and is tracked in #456: decimation on publication, volume.obj for the MaleCNS release, and gzip plus the contradictory cache-control: private on the data host. Decimation is the fix that makes these neurons genuinely viewable — everything here keeps them from breaking anything else in the meantime.