[BUG] PiperPhonemesToIdsVits (VITS + espeak data_dir) misses the pad token right after BOS, degrading pronunciation across the whole utterance

[forexdoctors-cmd]

Summary

PiperPhonemesToIdsVits() in sherpa-onnx/csrc/piper-phonemize-lexicon.cc
builds the token-id sequence for VITS/Piper models as:

[bos, p1, pad, p2, pad, ..., pn, pad, eos]

but the reference algorithm it is based on (and links to in its own
comment), rhasspy/piper-phonemize's phoneme_ids.cpp, produces:

[bos, pad, p1, pad, p2, pad, ..., pn, pad, eos]

i.e. a pad must also be inserted immediately after bos. This repo's own
generic OfflineTtsImpl::AddBlank() helper (in offline-tts-impl.cc)
confirms the same canonical pattern -- it wraps every token, including the
first one, in pad:

cppstd::vector<int64_t> buffer(x.size() * 2 + 1, blank_id);
int32_t i = 1;
for (auto k : x) {
buffer[i] = k;
i += 2;
}

However, AddBlank() is only invoked when
config_.model.vits.data_dir.empty() is true (see
OfflineTtsVitsImpl::Generate in offline-tts-vits-impl.h). Any model that
uses espeak-ng phonemization (i.e. sets --vits-data-dir, which is required
for every non-English Piper voice that needs G2P) goes through
PiperPhonemizeLexicon / PiperPhonemesToIdsVits instead, which has its
own inline padding logic -- and that logic omits the pad-after-bos step.

Net effect: for every Piper/VITS model loaded with --vits-data-dir (i.e.
essentially all non-English/non-"characters" Piper voices), every phoneme
from the very first one onward is fed to the model shifted by one position
relative to what it saw during training. The model was trained on data
produced by the canonical phonemize() + phonemes_to_ids() pipeline
(bos, pad, p1, pad, p2, ...), so this off-by-one is out-of-distribution for
the whole utterance, not just the first phoneme.

Reproduction

Custom-trained Piper VITS voice (Uzbek), exported via piper1-gpl's
export_onnx.py, metadata added manually to satisfy sherpa-onnx's
mandatory sample_rate/n_speakers/language metadata_props check.

Running the exact same phoneme sequence through the original
piper1-gpl Python inference pipeline (which correctly implements
bos-pad-p1-pad-...): correct pronunciation.
Running the same text through sherpa-onnx's
OfflineTts (VITS backend, --vits-data-dir set to espeak-ng-data):
audible degradation, most noticeably at word boundaries (e.g. "sh" /ʃ/
at the start of an utterance rendering closer to /h/, and word-final
consonants sometimes dropping in perceived clarity).
Verified independently that espeak-ng phonemization itself is correct
(built the exact frozen csukuangfj/espeak-ng@7251208 fork used by the
vendored piper-phonemize commit dc6b5f4441bffe521047086930b0fc12686acd56,
and confirmed --ipa output for the test sentence is correct: no bug in
the dictionary/rules/engine). No "Skip unknown phonemes" warnings are
logged either (all phonemes are present in tokens.txt), confirming the
discrepancy happens specifically in id-sequence construction, not in
phonemization or the token table.

Fix

In sherpa-onnx/csrc/piper-phonemize-lexicon.cc,
PiperPhonemesToIdsVits():

cpp ans.push_back(bos);
+ans.push_back(pad); // pad must follow bos, matching phoneme_ids.cpp
for (auto p : phonemes) {
if (token2id.count(p)) {
ans.push_back(token2id.at(p));
ans.push_back(pad);
} else {
SHERPA_ONNX_LOGE("Skip unknown phonemes. Unicode codepoint: \U+%04x.",
static_cast<uint32_t>(p));
}
}
ans.push_back(eos);

This is a one-line change and matches both the upstream piper-phonemize
reference implementation and this repo's own AddBlank() convention.

Happy to open a PR with this change if that's preferred over a maintainer
fix.

[dosubot]

Hi @forexdoctors-cmd! I'm Dosu and I'm helping the csukuangfj's Org team.

Your analysis is correct. I verified the upstream reference implementation in rhasspy/piper-phonemize/src/phoneme_ids.cpp and it clearly inserts a pad token immediately after BOS when interspersePad is enabled:

if (config.addBos) {
    auto const bosIds = &(phonemeIdMap->at(config.bos));
    phonemeIds.insert(phonemeIds.end(), bosIds->begin(), bosIds->end());
    if (config.interspersePad) {
        // Pad after bos (_)
        auto const padIds = &(phonemeIdMap->at(config.pad));
        phonemeIds.insert(phonemeIds.end(), padIds->begin(), padIds->end());
    }
}

Meanwhile, sherpa-onnx's PiperPhonemesToIdsVits function pushes bos and then immediately enters the phoneme loop without inserting pad first:

ans.push_back(bos);
for (auto p : phonemes) {
    if (token2id.count(p)) {
        ans.push_back(token2id.at(p));
        ans.push_back(pad);
    }
    ...
}
ans.push_back(eos);

This produces [bos, p1, pad, p2, pad, ..., pn, pad, eos] rather than the expected [bos, pad, p1, pad, p2, pad, ..., pn, pad, eos]. The comment in the sherpa-onnx code even references the upstream piper-phonemize implementation [1], making this an unintentional divergence rather than a design choice.

Your proposed one-line fix (ans.push_back(pad) after ans.push_back(bos)) aligns with both the upstream implementation and the repo's own AddBlank() helper logic [2]. A PR would likely be the cleanest path to getting this merged.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.