Skip to content

Commit 01914ba

Browse files
committed
test: add regression test for promptTokens with strict firstTokenLogProbThreshold
Verifies that transcription produces non-empty output when promptTokens are set, even with a strict firstTokenLogProbThreshold (-0.5). Before the fix, the prompt-shifted first token logprob would trigger the threshold and abort decoding immediately.
1 parent 5606dea commit 01914ba

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

Tests/WhisperKitTests/UnitTests.swift

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1702,6 +1702,34 @@ final class UnitTests: XCTestCase {
17021702
XCTAssertFalse(result.text.contains(promptText), "Prompt text should not be present in the result")
17031703
}
17041704

1705+
func testPromptTokensWithStrictFirstTokenThreshold() async throws {
1706+
// Regression test: promptTokens shift the decoder's KV cache state, which can
1707+
// cause the first content token's logprob to drop below firstTokenLogProbThreshold.
1708+
// With the fix, the threshold is skipped when promptTokens are present, so the
1709+
// decoder should still produce non-empty output even with a very strict threshold.
1710+
let config = WhisperKitConfig(model: "tiny", verbose: true, logLevel: .debug, load: true)
1711+
let whisperKit = try await WhisperKit(config)
1712+
let tokenizer = try XCTUnwrap(whisperKit.tokenizer)
1713+
1714+
let promptText = "繁體中文語音轉錄。"
1715+
let promptTokens = tokenizer.encode(text: promptText)
1716+
1717+
// Use a strict firstTokenLogProbThreshold that would have caused empty results
1718+
// before the fix (the prompt context shifts the first token's distribution).
1719+
let options = DecodingOptions(
1720+
skipSpecialTokens: true,
1721+
promptTokens: promptTokens,
1722+
firstTokenLogProbThreshold: -0.5
1723+
)
1724+
1725+
let result = try await XCTUnwrapAsync(
1726+
await transcribe(with: .tiny, options: options),
1727+
"Failed to transcribe"
1728+
)
1729+
1730+
XCTAssertFalse(result.text.isEmpty, "Transcription should not be empty when promptTokens are set, even with a strict firstTokenLogProbThreshold")
1731+
}
1732+
17051733
func testPrefixTokens() async throws {
17061734
let config = WhisperKitConfig(model: "tiny", verbose: true, logLevel: .debug, load: true)
17071735
let whisperKit = try await WhisperKit(config)

0 commit comments

Comments
 (0)