Skip to content

minor performance updates - #63

Merged
tpae merged 1 commit into
mainfrom
chore/update-dependencies-2
Sep 10, 2025
Merged

minor performance updates#63
tpae merged 1 commit into
mainfrom
chore/update-dependencies-2

Conversation

@tpae

@tpae tpae commented Sep 10, 2025

Copy link
Copy Markdown
Contributor

Summary

While investigating #47 , I noticed minor optimization to be had for Swift Jinja package.

Related PRs:
huggingface/swift-jinja#24
huggingface/swift-transformers#226

Before:

Server Model TTFT avg (ms) Total avg (ms) Chars/s avg TTFT rel Total rel Chars/s rel Success
Osaurus llama-3.2-3b-instruct-4bit 89 1325 553 0% 0% 0% 100%
Ollama llama3.2 59 1669 439 +33% -26% -20% 100%
LM Studio llama-3.2-3b-instruct 56 1159 618 +37% +13% +12% 100%

After:

Server Model TTFT avg (ms) Total avg (ms) Chars/s avg TTFT rel Total rel Chars/s rel Success
Osaurus llama-3.2-3b-instruct-4bit 86 1314 558 0% 0% 0% 100%
Ollama llama3.2 58 1655 434 +32% -26% -22% 100%
LM Studio llama-3.2-3b-instruct 56 1203 610 +34% +8% +9% 100%

Although the benchmark performance seems very minor (maybe 3% closer to LM Studio), Jinja optimization should have lasting effects beyond the benchmarks.

I will be getting back to the drawing board to find some other areas for improvements. ✏️

Changes

  • Behavior change
  • UI change (screenshots below)
  • Refactor / chore
  • Tests
  • Docs

Checklist

  • I have read CONTRIBUTING.md
  • I added/updated tests where reasonable
  • I updated docs/README as needed
  • I verified build on macOS with Xcode 16.4+

@tpae
tpae merged commit 08f9bed into main Sep 10, 2025
@tpae tpae added the released label Sep 12, 2025
tpae added a commit that referenced this pull request Mar 8, 2026
@tpae
tpae deleted the chore/update-dependencies-2 branch March 31, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant