updated applyChatTemplate with lazy memoization

[tpae]

Based on #217.

Cache compiled Jinja Template instances in PreTrainedTokenizer keyed by the template string; reuse in applyChatTemplate. No API or behavior changes.

Measured performance: Repeated calls with the same template are ~6x faster.

Cached (reused template): ~0.00049 s (0.49 ms) average
Uncached (unique template each call): ~0.00292 s (2.92 ms) average
Speedup: ~5.9–6.2x

Raw results:

Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' started.
swift-transformers/Tests/TokenizersTests/ChatTemplateTests.swift:276: Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 7.781%, values: [0.000339, 0.000385, 0.000339, 0.000385, 0.000316], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' passed (0.986 seconds).
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' started.
swift-transformers/Tests/TokenizersTests/ChatTemplateTests.swift:287: Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' measured [Clock Monotonic Time, s] average: 0.003, relative standard deviation: 2.207%, values: [0.002790, 0.002796, 0.002700, 0.002880, 0.002734], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' passed (0.932 seconds).

[FL33TW00D]

@tpae great contribution thank you.

If you could format it with the pinned version of swiftformat we can merge!

brew install swiftformat@0.54.6
swiftformat --config .swiftformat .

[pcuenca]

Looks very nice, thank you @tpae! 🙌

[tpae]

Amazing! Thank you!

My brew couldn't find swiftformat@0.54.6, but was able to find 0.57.2, and I ran the formatting.

[tpae]

Test was passing for me locally 🤔 i think 429 is rate limiting

[pcuenca]

Tests pass for me locally as well, and the CI worked for this PR too. So I think the token may not be passed to the CI from forks. I'll take a quick look, otherwise I'll merge and resolve later.

[pcuenca]

I'm merging. Also opened #222 which could be interesting for actual use cases, not just the CI.

[pcuenca]

Thanks a lot @tpae!