Use the key ideas from Transcoda: Synthetic datasets and better data augmentations

[liebharc]

AI Disclaimer: Another run which largely relied on Claude Code

The transcoda paper: https://arxiv.org/pdf/2605.10835

The synthetic approach is roughly this:

• Take a dataset and parse it into the internal format (tokens in our case)
• From the internal dataset, render an image (in our case tokens->musicxml->image)
• You now have an image which most likely matches exactly the ground truth data

This approach has the downside that the images become perhaps too clean. Everything which is nor part of your internal format will be removed. Transcoda therefore injects certain symbols again.

Suggestion: We already have a pretty clean pipeline build on Lieder, Primus and grandstaff. I think we should continue using that to have some more noisy examples (meaning examples with symbols which are not in our internal format and which use a different music font/renderer). And that we only add the Transcoda approach for the two new dataset musetrainer (rather tiny) and PDMX (large).

[aicelen]

What do you think about using the existing Lieder dataset pipeline for Pdmx as well?

I had a few more ideas after taking a quick look at the code:

• we could use this PRs data augmentation for the segnet as well
• we could also augment the whole score and run it through homr's preprocessing to get staffs similar to what the transformer "sees" during inference
• it seems like musescore also supports multiple "fonts"

[liebharc]

What do you think about using the existing Lieder dataset pipeline for Pdmx as well?

All your ideas sound good to me. I think each of them could work out. The PDMX dataset seems to be large enough that we could also split it and use both approaches (lieder & synthetic). So there is a lot of room for experimentation.

However, we should base experiments on numbers/benchmarks to really find out what works better. And then go step by step to not get carried away. E.g. also for this PR I think it's better to first do a run without the new datasets to just see the impact of the improvements to the lieder dataset (other PRs) and only then add the new datasets to a training run. Otherwise, we will have no idea about the individual contributions to the final model.

[liebharc]

This would be ready to merge now. The new datasets are disabled until we decide to use them. As mentioned above, it might be worthwhile to run them through the lieder pipeline instead. Next to the datasets the PR add:

• Faster musicxml generator
• Fixes for the musicxml parser
• Updated data augmentation, which overall seems beneficial - full training run needed to confirm that

[liebharc]

A 10 epoch run with lieder+grandstaff+primus, so without the new datasets

## Run 423 at epoch 10

Commit: f7bef4eb359a11611048155e048d1a65e907e96e
Day: 9 July 2026
Transformer Smoke Test: 10%
System Level: 7.3 diffs, SER: 5.3%
Polish scores:  26.85%

[liebharc]

A 8 epoch run with lieder+grandstaff+primus+pdmx+musetrainer

## Run 426 at epoch 8 which is about the same training time as 423 at epoch 10

Commit: b6fd20809a8dcaf10dfd39a4ca4f64c6f056e644
Day: 10 July 2026
Transformer Smoke Test: 10%
System Level: 6.0 diffs, SER: 4.8%
Polish scores:  25.65%

[liebharc]

Some more notes how the different epochs perform. Not much new here, it only takes 10 epochs to get decent results, but then at least another 10 epochs is needed to get to the performance we aim for.

./checkpoint-56384/model_ser.txt epoch 8
Transformer Smoke Test: 10%
System Level: 6.0 diffs, SER: 4.8%
Polish scores:  25.65%

./checkpoint-63432/model_ser.txt epoch 9
10%
8.5 diffs, SER: 8.9%

./checkpoint-70480/model_ser.txt epoch 10
8%
6.0 diffs, SER: 4.7%
25.63%

./checkpoint-77528/model_ser.txt epoch 11
9%
5.9 diffs, SER: 4.4%
Polish scores: 26.18%

./checkpoint-84576/model_ser.txt epoch 12
8%
27.6 diffs, SER: 10.1%

./checkpoint-91624/model_ser.txt epoch 13
8%
6.3 diffs, SER: 5.1%


./checkpoint-98672/model_ser.txt epoch 14
7%
5.6 diffs, SER: 4.5%
25.46%

./checkpoint-119816/model_ser.txt epoch 17
6%
5.8 diffs, SER: 4.8%

./checkpoint-126864/model_ser.txt epoch 18
9%
5.5 diffs, SER: 4.5%

./checkpoint-133912/model_ser.txt epoch 19
8%
5.6 diffs, SER: 4.5%

./checkpoint-162104/model_ser.txt epoch 23
6%
5.3 diffs, SER: 4.2%

./checkpoint-190296/model_ser.txt epoch 27
Transformer Smoke Test: 5%
System level: Total: 5.5 diffs, SER: 4.1%
Polish scores: OMR-NED 24.30%

[weixlu]

Hi, I notice this PR has been merged. Cheers! It seems the new model 426 hasn't been tested on smb yet. Let me do it.

[weixlu]

Benchmark on smb takes about 2 hours to run on my machine. I'll profile this and see if there is a chance to speed up

[liebharc]

Hi, yes I thought it's better to merge this now as it was open for a long while, and it accumulated some larger changes. It seems to improve the overall performance, but just by a bit. But it can be a starting point for further experiments:

• Use some of the ideas @aicelen described earlier
• Use a curriculum learning approach and e.g. start with the cleaner synthetic samples first
• Try variations of the model parameters, e.g. larger convnext, convnextv2, deeper decoder

Practically I feel like the training time is at the limit of what I can do on a single GPU as 426 took 3.5 days.

But back to goals, the "Transformer Smoke Test" and "System Level" metrics are likely in a range now where you can't see model improvements anymore. The datasets are small and the remaining errors tiny so that any changes you see there are more likely noise than real. Doing any changes to the training and model without guiding metrics would be just guessing. So the next logical step would be to look at both benchmarks and see what issues we can spot there and then use that as a starting point to draft ideas on how we could further improve.

[weixlu]

OK, I got it. This may explain why you introduce the benchmark, since we already have Smoke Test & System Level Test.

You know that I'm always happy to do the training work. If you already have done some experiments, just push the branch and I'll start training. That's no burden to me at all.

[weixlu]

You mentioned three possible directions to explore, and I’ll take a look at them. To be honest, though, I feel that my background is more in engineering than in research. For example, I saw that this PR was inspired by a paper, but I didn’t really understand it. For this PR, I just find out how to run, but not yet look closely into its principles.

[liebharc]

I'm also an engineer not a researcher. I can work through papers, but most of the time the source code is clearer to me. And thanks again for the offer to do training runs. It's really appreciated!

Regarding the ideas: I'm not sure if it's worth to follow up on them right now. Likely the hard work to go through error cases (if any new issues come in) or benchmarks is the better approach (while being careful to not solely optimize for the benchmark).

[weixlu]

Thanks for your detailed explanation! So there are many directions to explore. For now my work is just some small fix or optimization, which can be done in a day. Later when I do something bigger, I think I'll open a issue for further discussion.