Allow overriding the mlx.metallib search path for plugin-bundle embedding

[olilarkin]

Summary

There is currently no public way to tell MLX where to find mlx.metallib at runtime. When MLX is embedded inside a plugin bundle (e.g. an Audio Unit or VST3), the host application owns the main bundle, so MLX's default metallib search paths (colocated / SwiftPM resource bundle / main bundle) don't resolve and Metal initialization fails.

Proposal

Add an API to set an explicit mlx.metallib path before the Metal device initializes. If set, it is tried first and falls back to the existing search logic if loading fails — so default behavior is unchanged when unset.

This spans three layers:

• mlx (C++) — mlx::core::metal::set_metallib_path(const std::string&)
• mlx-c (C) — mlx_metal_set_metallib_path(const char*)
• mlx-swift (Swift) — GPU.setMetallibPath(_ path: String)

Use case

Running MLX-Swift inside AU/VST3 audio plugins, where the standard bundle-relative search paths point at the host app rather than the plugin.

PRs

Implemented as a dependency chain (C++ → C → Swift). The submodule bumps in the mlx-swift PR should be re-pointed to the merged upstream commits once the lower PRs land.

1. Add metal::set_metallib_path() to override the metallib search path mlx#3597 — C++ set_metallib_path
2. Add mlx_metal_set_metallib_path() C API mlx-c#117 — C API wrapper
3. Add GPU.setMetallibPath() for plugin bundle support #416 — Swift GPU.setMetallibPath()

[olilarkin]

Is in-process plugin embedding actually in scope for MLX?

Before this moves forward I want to raise a more fundamental concern, because the metallib path is really just the surface symptom.

The motivating use case is running MLX inside an AU/VST3 audio plugin. The deeper issue is that MLX is built on process-global singletons — the default device, the Metal device (mlx/backend/metal/device.cpp), the MetalAllocator (mlx/backend/metal/allocator.cpp), and the scheduler. For in-process plugins that splits into two cases:

• N instances of one plugin sharing a single MLX runtime — this is fine, even desirable: one shared GPU memory pool and kernel cache rather than N. The recent thread-safety hardening (evalLock, clearCache / wired-memory race fixes) supports this.
• Two copies of MLX in one host process — via a diamond dependency, or the case I genuinely can't control: a different vendor's plugin loaded into the same host also linking MLX, possibly a different version. That gives two MetalAllocator singletons contending for VRAM and duplicate-symbol / ODR hazards.

Mitigations exist for my own build (link MLX as a single shared framework with hidden symbol visibility / two-level namespace, set global config once at startup, keep all MLX calls off the realtime audio thread). But the "another plugin also embeds MLX" case is a soft failure mode I can't fully design away.

So my honest question to the maintainers: is embedding MLX in in-process plugins a configuration you consider supported? If it fundamentally isn't, that's a perfectly reasonable answer.

On the PRs (ml-explore/mlx#3597, ml-explore/mlx-c#117, #416): setMetallibPath() isn't actually plugin-specific — it's an opt-in override for any case where the default colocated/bundle search doesn't resolve (custom packaging, relocated resources, CLI tools shipping the metallib elsewhere), and it's a no-op when unset. So it arguably has standalone value even if in-process plugins are out of scope. But if you don't see value beyond the plugin case, feel free to close — no objection here.