Benchmarks and smol LLM model

[bevenky]

Has this been compared with livekits turn detection model that uses smolllm? And would using smolllm makes this any faster?

[kwindla]

LiveKit didn't release any benchmarks or training data and the license for that model doesn't allow using it outside the LiveKit Agents framework, so to do benchmarks you'd need to set something up using the LiveKit tech stack and then run something like the human_5 data set in this repo through both models.

An audio model can classify based on acoustic information like a rising intonation at the end of a segment, drawing out a word at the end of a segment, etc. Using an audio model also lets you train for this problem end-to-end, rather than being limited by whatever the transcription model you’re using does/doesn’t do.

SmolLM is definitely faster. It's a smaller model. I think we can optimize the Wav2Vec2-BERT inference a lot, though. There's a lot of platform-specific low-hanging fruit (flash attention, coreml). And, with more work, my guess is that we can get down to 100ms inference on CPU.

[ChenghaoMou]

If you are looking for a smolLM alternative, check out https://github.com/latishab/turnsense

I am doing a text-based semantic VAD benchmark based on https://huggingface.co/datasets/latishab/turns-2k and https://github.com/TEN-framework/ten-turn-detection, will share the results when ready.

[ChenghaoMou]

Eval set: ten-turn-test
ten results:
              precision    recall  f1-score   support

           0       0.94      0.95      0.95       426
           1       0.93      0.95      0.94       508
           2       1.00      0.92      0.96       200

    accuracy                           0.94      1134
   macro avg       0.96      0.94      0.95      1134
weighted avg       0.94      0.94      0.94      1134

turnsense results:
              precision    recall  f1-score   support

           0       0.60      0.76      0.67       426
           1       0.57      0.67      0.61       508
           2       0.00      0.00      0.00       200

    accuracy                           0.59      1134
   macro avg       0.39      0.48      0.43      1134
weighted avg       0.48      0.59      0.53      1134

Eval set: turns2k
ten results:
              precision    recall  f1-score   support

           0       0.87      0.96      0.92       884
           1       0.98      0.88      0.93      1116
           2       0.00      0.00      0.00         0

    accuracy                           0.92      2000
   macro avg       0.62      0.62      0.61      2000
weighted avg       0.93      0.92      0.92      2000

turnsense results (overestimation since this overlaps with its training dataset):
              precision    recall  f1-score   support

           0       0.97      0.89      0.93       884
           1       0.92      0.98      0.95      1116

    accuracy                           0.94      2000
   macro avg       0.94      0.93      0.94      2000
weighted avg       0.94      0.94      0.94      2000

Corresponding confusion matrices:

It seems that Ten turn detection generalizes well in this case, but it is also a 7B model so no surprise there. turnsense has a disadvantage for Chinese so it performs relatively bad on ten's test set.

[marcus-daily]

We've now released Smart Turn v2, which is smaller in size (95M params) and so should be faster than SmolLM-based models, while still using audio-native input for greater accuracy.

v2 is also multilingual with support for Chinese: https://www.daily.co/blog/smart-turn-v2-faster-inference-and-13-new-languages-for-voice-ai/

[marcus-daily]

Smart Turn v3 is now released, with int8 quantization, and a base model change (to Whisper Tiny). v3 is up to 100x faster than v2 with the same accuracy, and can run efficiently on CPU for the first time!

https://www.daily.co/blog/announcing-smart-turn-v3-with-cpu-inference-in-just-12ms/