perf(linux): trim glibc malloc arena after module loading

[bartlomieju]

Summary

• Calls malloc_trim(0) on Linux after the main/side module loading phase completes, forcing glibc to release unused heap pages back to the OS
• Addresses 3-5x higher RSS on Linux vs Windows when loading large TypeScript codebases (e.g. generated OpenAPI clients)
• Module compilation creates heavy allocation churn that glibc's allocator retains in its arena; malloc_trim reclaims this

Root cause

glibc's malloc doesn't automatically return freed memory to the OS — it keeps freed pages in its arena for future reuse. On Windows, the NT heap manager coalesces and releases freed memory much more aggressively. After TypeScript compilation/module loading (which creates many temporary allocations for AST nodes, source maps, etc.), glibc holds onto all that memory even though it's been freed.

Why this is safe

• malloc_trim is a standard glibc function, safe to call at any time
• It only releases memory already freed — no impact on live allocations
• Guarded by #[cfg(target_os = "linux")] — no-op on other platforms
• The existing DENO_USR2_MEMORY_TRIM mechanism already uses the same call, just triggered externally via SIGUSR2

Closes #25722

Test plan

• Verify compilation succeeds on Linux (cargo check -p deno)
• Run spec tests (cargo test specs)
• Test with a large TypeScript project on Linux and measure RSS before/after with Deno.memoryUsage().rss
• Verify no regression on Windows/macOS (the malloc_trim call is cfg-gated to Linux only)

🤖 Generated with Claude Code