Skip to content

Add SmolLM2-1.7B-Instruct support (Apache 2.0, 1.7B) - #76

Closed
stikves wants to merge 1 commit into
apple:mainfrom
stikves:sukru/smollm2-text
Closed

Add SmolLM2-1.7B-Instruct support (Apache 2.0, 1.7B)#76
stikves wants to merge 1 commit into
apple:mainfrom
stikves:sukru/smollm2-text

Conversation

@stikves

@stikves stikves commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Add HuggingFace SmolLM2-1.7B-Instruct for on-device inference. Apache 2.0 licensed, standard Llama architecture.

Changes

  • Add llama model_type to registry (maps to existing MistralForCausalLM)
  • Add smollm2-1.7b-instruct preset (INT4 default)
  • models/smollm2/README.md — Usage documentation

Architecture

SmolLM2 uses standard Llama architecture — no new model code needed:

  • 24 layers, 2048 hidden, MHA (32/32 heads), head_dim=64
  • SiLU MLP, RMSNorm, standard RoPE (theta=130000)
  • tie_word_embeddings = True
  • Vocabulary: 49K tokens

Performance (Apple Silicon, INT4)

Metric Value
Prefill ~502 t/s
Generation ~176 t/s
Model Size ~1 GB

Export

uv run coreai.llm.export HuggingFaceTB/SmolLM2-1.7B-Instruct

Test plan

  • Model loads via existing MistralForCausalLM class
  • Export completes (INT4)
  • E2E inference produces correct output ("The capital of Japan is Tokyo")
  • 176 t/s generation speed

HuggingFace SmolLM2 for on-device inference. Apache 2.0 licensed,
standard Llama architecture (reuses existing mistral model class).

- Add llama model_type to registry (maps to MistralForCausalLM)
- Add smollm2-1.7b-instruct preset (INT4 default)
- 176 t/s generation on Apple Silicon (INT4)
- models/smollm2/README.md with usage docs
@stikves stikves closed this Jul 2, 2026
@stikves
stikves deleted the sukru/smollm2-text branch July 2, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant