Preserve Punctuation in Alignment Output

[shanureddy4]

Hi,

I am trying to align a transcript file (.txt) with an audio file (.wav). The alignment works well, and the words match perfectly with correct casing. However, I noticed that punctuation marks (such as commas and periods) are missing in the output.

For example, I expect to get "Together," instead of "Together". But in the alignment result, punctuation is not included:

{
  "type": "sentence",
  "text": "Together, they worked towards a brighter future, one where humanity could thrive once more.",
  "startTime": 80.8748125,
  "endTime": 85.59931250000001,
  "timeline": [
    {
      "type": "word",
      "text": "Together",
      "startTime": 80.8748125,
      "endTime": 81.3575
    }
  ]
}

Here’s how I am calling the alignment function:

const options = {
    language: 'eng'
};

console.log('Starting alignment process...');

// Perform the alignment
const result = await Echogarden.align(audioPath, transcript, options);

I checked the available interfaces but couldn't find an option to preserve punctuation. Could you please guide me on how to retain punctuation in the alignment result?

Thank you!

[shanureddy4]

As I observed in the CLI, the alignment is able to generate word-by-word output directly from the transcript file while preserving the case, commas and peroid. I am trying to achieve the same result, but not in real-time (CLI giving in real time).

[rotemdan]

The word entries in the timeline don't include punctuation. There are several reasons for that:

• The timeline was originally designed for text-to-speech word highlighting, where highlighting punctuation is usually not desirable
• Word entries have phoneme timelines, so including punctuation within words would not be very suitable
• The goal of the timeline is also to provide word segmentation and word to phoneme conversion data, so punctuation isn't desirable for that
• The alignment processing concentrates on pronounceable words as part of how it works internally - it actively tries to filter out anything else

The timeline could technically include independent entries for punctuation - basically "word" entries dedicated for punctuation sequences between words. It's not clear if such entries would include whitespace, like spaces, tabs, and line breaks (should a single space between words have its own entry?). If they are added as an option, I think it would be useful to annotate them in some way, and not give them a type word, but maybe punctuation-word or something like that.

It would make the timeline larger, in any case, and I got people saying they would prefer it to be smaller overall.

I'll take adding an option this as feature suggestion, but it's going to have a low priority (I'm not doing much development at this time in general).

Anyway, for now, the timeline contains word start and end offsets (for both utf-8 and utf-16). The end offset of a word is where trailing punctuation would start. You can add the punctuation in code by iterating any punctuation characters, starting at that offset, in the original text string.

That's the way the CLI does it: the CLI shows the trailing punctuation after words by iterating the characters after the word and adding any punctuation characters that follow it. It doesn't use the timeline for that.

[shanureddy4]

I've been checking this thread since yesterday and, after debugging the CLI code, I couldn't figure out how you managed it. I feel a bit silly for not realizing that trailing punctuation would start after the end offset. Huge thanks to you for your project—it's really been a big help!