Skip to content

PERF/mobile: Reduce Nemotron load-time memory peak #12

Description

@MikeSchirtzinger

Problem

Nemotron's encoder is materialized as a full JavaScript Uint8Array before ORT session creation. During session build, this likely doubles the largest model allocation and makes mobile/lower-memory browsers much more likely to kill the tab.

Evidence

  • nemotron-engine.js:180-190 fetches encoder/decoder/tokenizer bytes, then passes them into WasmNemotron.create(enc, dec, tok).
  • nemotron-engine.js:383-400 streams the encoder for progress, but accumulates the full file into new Uint8Array(len).
  • crates/nemotron-asr/src/backend_web.rs:159-204 constructs ORT sessions from in-memory bytes.

Acceptance criteria

  • Investigate whether ORT web/session creation can avoid a second full encoder copy.
  • If possible, stream or transfer bytes into the WASM/worker side to lower peak memory.
  • If not possible with current ORT APIs, document the hard blocker and add a memory guard that routes low-memory devices away from Nemotron before loading.
  • Measure peak memory before/after or provide a reproducible browser memory trace.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions