History dropout

[liebharc]

Replaces scheduled sampling with history dropout to force the transformer to rely more on image data and less on the sequence history.

E.g. in some examples the transformer start to ignore the image and completely hallucinated tokens. Likely because it learned some typical sequence patterns during training.

@weixlu, could I ask you to do a training run with this branch? A quick 4 epoch run looks okay (SER 17%, system level: 10.7 diffs, SER: 7.6%). To understand if the issue is truly resolved we would unfortunately need a full run.

[weixlu]

Sure, no problem. I have started the training run already.

This is my first training run after #107 . Could you confirm whether the dataset size is around 250k?

Total number of training files to choose from 254628

The reason I ask is that when I was converting the two new datasets, I noticed many SIGSEGV messages in the logs. In fact, I just finished investigating this issue and was about to submit a PR to fix it.

[weixlu]

And this time it is running at around 2.5s/it, while previously it was about 1s/it, so it looks slower.

[liebharc]

The numbers roughly match, I have 254096. We also only use a small subset of pdmx and I don't think adding more will be necessarily beneficial. So I was just fine with the SIGSEGV errors. But if you fixed them, then that's good as well.

If anything training should actually be a bit faster not slower. I get 1.42s/it right after the training start on a single GPU setup. Starting training on main gives around the same value, so the training speed appears unchanged.

[weixlu]

If anything training should actually be a bit faster not slower

That's a little strange, let me take a closer look

[aicelen]

We also only use a small subset of pdmx and I don't think adding more will be necessarily beneficial.

Why, doesn't Transcoda use like 300k examples?

[liebharc]

Why, doesn't Transcoda use like 300k examples?

You mean, why don't we use more of PDMX samples like Transcoda does? So far training with PDMX showed only a minor performance improvement. I would have expected more just alone from having a larger variety of music fonts and samples.

[aicelen]

Ok, I thought that we use way more Pdmx training data while also trying to make the model larger. If you think this is worth the time I'd try this later.

[liebharc]

Possibly not sure. Not seeing a performance improvement despite the larger data set could be explained by model limitations. Just in the past that was usually the wrong lever. Well again, experience can be a bad guide. You can simply increase _TARGET_FILES in convert_pdmx or remove the limit entirely to get more out of the dataset.

As more data increases the training time, one might also consider a curriculum learning approach. But have I have no experience with that (well no experience, is also a bad guide, isn't it?).

[weixlu]

Some quick update on training: The training on my side still looks not good. It's slow when I use 4 GPUs, and it hangs when I use only 1 GPU. I use some tools to see where it hangs, it turns out to be torch.dataloader

When I checkout commit 4d3826 , the training goes normally, but the next commit won't work. So I think it has something to do with the 2 new datasets. (I tried removed the whole dataset and re-generate it again, but doesn't work)

It's 23:30 pm here in China so I have to go to bed. Sorry for the dalay, I'll go further if I have time on weekends, or maybe leaving it to next week.

[liebharc]

@weixlu really don't worry. This is a hobby project. Don't put yourself under pressure. Your help is really appreciated and things have time.

[weixlu]

You are so kind. No worries, I’m not really under much pressure, I just want things work as expected.

Finally, I managed to get the training run. Still need help from Claude Code 😄, I myself not so familiar with this part. Adding a few config was enough to fix it.

set -x NUMBA_THREADING_LAYER workqueue
set -x OMP_NUM_THREADS 4 (or 1)
cv2.setNumThreads(0) in training/transformer/image_utils.py

I’m not sure whether others will encounter the same issue, but Claude said this deadlock (hang) is theoretically possible to happen anywhere. In my case, the CPU is 128 cores, there are so many threads so as to make it guaranteed to happen.

[liebharc]

@weixlu thanks for the training run, the metrics are basically unchanged:

## Run 433

Commit: 5d4c0a1f74d2122c28f69fa951e42ad4d10bb0cf
Day: 20 July 2026
Transformer Smoke Test: 5%
System Level: Total: 5.0 diffs, SER: 4.3%
Polish scores: OMR-NED 24.36%
SMB scores: OMR-NED 22.28%

Did you in the end notice a difference in training time?

[weixlu]

not really, I think it's nearly the same as before

[liebharc]

ok, then I don't think there is a reason to merge this. Thanks for doing the training run!