batched-wav-nnet3-cuda2 and batched-wav-nnet3-cuda-online : online CPU feature extraction with pitch when decoding

[garymmi]

batched-wav-nnet3-cuda2 and batched-wav-nnet3-cuda-online both use the same online cpu feature extraction code
reference void BatchedThreadedNnet3CudaOnlinePipeline::ComputeOneFeature(int element) in cudadecoder/batched-threaded-nnet3-cuda-online-pipeline.cc

the code do cpu feature extraction chunk by chunk.
if use batched-wav-nnet3-cuda command , i got a total 366 frames for a test utterance.
if i use batched-wav-nnet3-cuda2 , i print frames of each chunk , and sum all, got 289 frames.
this is because pitch feature extraction has latency
the code also no do feature_pipeline.InputFinished(), so cannot got total 366 frames.

hope someone can help fix it, thanks

[danpovey]

Is that because InputFinished() is not implemented for CUDA, or because it's not called in the binary?

[garymmi]

reference this, because use cpu to do feature extraction, so use OnlineNnet2FeaturePipeline to do feature extraction
finally no do InputFinished() to flush remained frames

[garymmi]

if do InputFinished() at last chunk, nframes > 51, get assertion
KALDI_ASSERT(ninput_frames <= input_frames_per_chunk_);

[danpovey]

mm OK. Do you think you could add some print statements to the CUDA and CPU versions and see where the difference crept in, i.e. why one crashes and the other not?

…

On Fri, Dec 4, 2020 at 12:54 PM garymmi ***@***.***> wrote: if do InputFinished() at last chunk, nframes > 51, get assertion KALDI_ASSERT(ninput_frames <= input_frames_per_chunk_); <https://github.com/kaldi-asr/kaldi/blob/master/src/cudadecoder/batched-static-nnet3.cc#L160> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO3CT4TQLDKOUJ7UW5LSTBTORANCNFSM4UM354GQ> .

[dgxlsir]

when I run my script(use batched-wav-nnet3-cuda2 to decode my chain_model), some audio tail doesn't recognized,(1 char or 2 chars lose). I tried to adjust '--max-count',but got no results!
Best regards!

[garymmi]

mm OK. Do you think you could add some print statements to the CUDA and CPU versions and see where the difference crept in, i.e. why one crashes and the other not?

if no pitch, both cpu and gpu's feature extraction can get 366 frames
cuda's feature extraction, no implement pitch , so can get each chunk <= 51 frames

for example, normally frames of each chunk show as below , note each number of frames are less or equal 51
49
51
51
51
51
51
51
11
total frames are 366
if add pitch(cpu fe), frames of each chunk's
0
23
51
51
51
51
51
11
total frames are 289

if do InputFinished at last chunk
0
23
51
51
51
51
51
88
because the frames of last chunk is 88 large 51, get assertion
reference this code KALDI_ASSERT(ninput_frames <= input_frames_per_chunk_);

[danpovey]

I didn't write that code and am not 100% sure how to fix. It looks like there is something 'flush_eos_context' which is supposed to handle this. Is that set to true? Perhaps there is a different interface for this code.

…

On Fri, Dec 4, 2020 at 3:26 PM garymmi ***@***.***> wrote: mm OK. Do you think you could add some print statements to the CUDA and CPU versions and see where the difference crept in, i.e. why one crashes and the other not? if no pitch, both cpu and gpu's feature extraction can get 366 frames cuda's feature extraction, no implement pitch , so can get each chunk <= 51 frames for example, normally frames of each chunk show as below , note each number of frames are less or equal 51 49 51 51 51 51 51 51 11 total frames are 366 if add pitch(cpu fe), frames of each chunk's 0 23 51 51 51 51 51 11 total frames are 289 if do InputFinished at last chunk 0 23 51 51 51 51 51 88 because the frames of last chunk is 88 large 51, get assertion reference this code KALDI_ASSERT(ninput_frames <= input_frames_per_chunk_); <https://github.com/kaldi-asr/kaldi/blob/master/src/cudadecoder/batched-static-nnet3.cc#L160> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO5C4WGCM64JY6B7V33STCFLZANCNFSM4UM354GQ> .

[hugovbraun]

I see, it's because when called with "InputFinished" we get too many frames (the rest of the pipeline can support only up to input_frames_per_chunk_ frames). We weren't calling "InputFinished" which probably explains @dgxlsir's bug. That can be fixed.

Regarding the 366 vs 289 frames, is the difference explained only by that "InputFinished" issue? It looks like a large difference between the two.

[garymmi]

i did the test
Model : mfcc + pitch + ivector
Config : --use-online-features=true --gpu-feature-extract=false

because pitch cannot synchronously generated with mfcc, reference feat/pitch-functions.cc#L1577, it must wait more frames to input.
so feature extraction loss frames in batched-wav-nnet3-cuda2 and batched-wav-nnet3-cuda-online if pitch enable

[hugovbraun]

Ok, thanks for the details. We'll fix this early January. It should be fairly simple to fix at the dynamic batcher level.

[superliuwen]

Hi @hugovbraun , I also meet this issue, when pitch is used in cuda decoder, the cer is much higher. Do you have a plan to solve this issue? Thanks a lot!