Skip to content

SpeakerKit: The last turn is attributed to the wrong speaker #485

Description

@robertobrega

File: Sources/SpeakerKit/Pyannote/SpeakerSegmenterModel.swift
(predict(...), in the per-chunk worker, just before
AudioProcessor.padOrTrimAudio).

Symptom. The last ~10 s of a recording
are reproducibly split off into a phantom extra speaker, even for a
single-voice recording on one channel.

Root cause. The segmenter's CoreML model has a fixed [480000]
(30 s) input, so the final short chunk is zero-padded to 30 s
(padOrTrimAudio). The embedder then runs its preprocessor over the
whole padded 30 s chunk (SpeakerEmbedderModel.processChunk,
SpeakerEmbedderPreprocessorInput(waveforms:)). A global op in that
frontend lets the trailing zeros attenuate the embeddings of the
chunk's real-audio windows
— their L2 norm roughly halves
(~3.7 → ~2.1, and as low as ~1.1) even when the window's speaker mask
lies entirely inside real audio. Because the tail windows of a recording are only ever
emitted from this padded final chunk, the last paragraph gets the
phantom speaker almost every time.

Fix. Fill the final short chunk's tail by mirror-reflecting the
real audio
(alternating reversed/forward copies) instead of
zero-padding. This keeps the preprocessor's global statistics in a
speech regime, so the real-window embeddings keep their normal norm and
stay in the correct cluster. waveformLength is still computed from the
real sample count, so bounded() and downstream turn-clamping are
unchanged — only the model's input padding content differs.

The diff
Two changes, both in SpeakerSegmenterModel.swift:

  1. New reflect-pad block:

var srcWaveform = chunk.waveform
if srcWaveform.count < maxChunkLength, !srcWaveform.isEmpty {
var ext = srcWaveform
var forward = false // alternate reversed / forward copies = mirror reflect
while ext.count < maxChunkLength {
let block = forward ? srcWaveform : Array(srcWaveform.reversed())
ext.append(contentsOf: block.prefix(maxChunkLength - ext.count))
forward.toggle()
}
srcWaveform = ext
}

  1. One-line redirect — feed the reflected buffer into the model instead of the raw chunk:

guard let audioSamples = AudioProcessor.padOrTrimAudio(
fromArray: chunk.waveform,
fromArray: srcWaveform,
startAt: 0,
toLength: maxChunkLength
) else {

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions