Expand Preset Catalog for Lightweight On-Device Models & Audio Tasks

[omen18]

Feature Request: Expand Preset Catalog for Lightweight On-Device Models & Audio Tasks

Feature Description

We propose extending LLM_PRESETS and UtilityModel entries in model_registry.py to support popular small-footprint open-weight models optimized for Apple Silicon on-device execution.

Proposed Model Additions

1. Small LLM / Speculative Decoding Draft Models

• Qwen 2.5 0.5B Instruct (Qwen/Qwen2.5-0.5B-Instruct): Extremely fast draft model for speculative decoding on iOS/macOS.
• Llama 3.2 1B Instruct (meta-llama/Llama-3.2-1B-Instruct): Lightweight mobile text assistant.

2. Audio & Speech Models

• Whisper Tiny / Base (openai/whisper-tiny, openai/whisper-base): Low-latency streaming speech-to-text models for real-time applications.

Value Proposition

Adding these presets allows developers to export lightweight draft models and on-device speech models via uv run coreai.model.registry without needing custom --experimental flags or manual parameter tuning.