Filtering empty text examples

[artemru]

What does this PR do? Please describe:
Removing empty text examples in ASR dataloaders

• This requires to move tokenization before bucketing!

Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.

Check list:

• Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
• Did you read the contributor guideline?
• Did you make sure that your PR does only one thing instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

[mattsetz]

looks good! do we expect this will slow down dataloading time to tokenize examples one by one? or maybe its minimal bc the tokenization is so fast?

[mattsetz]

we can add another arg here remove_unk to optionally remove all UNKs tokens from transcript. we should default this to False but it can be something that we experiment with.

[mattsetz]

also, maybe we can log the number of UNKs (or % of num_unks/total_tokens within a sample/batch) in weights and biases? its useful for us to know if our dataset contains too many UNKs.

[artemru]

also, maybe we can log the number of UNKs (or % of num_unks/total_tokens within a sample/batch) in weights and biases? its useful for us to know if our dataset contains too many UNKs.

it's possible (requires a bit to propagate some metric bags to data reader) but for now we can just do in a adhoc analysis

[artemru]

we can add another arg here remove_unk to optionally remove all UNKs tokens from transcript. we should default this to False but it can be something that we experiment with.

done !

[artemru]

looks good! do we expect this will slow down dataloading time to tokenize examples one by one? or maybe its minimal bc the tokenization is so fast?

there'll not be slowdown in the iterations (still need to do tokenization just in different order), but it will slightly increase the starting time. Tokenization is rather fast (>200k tokens /sec)