Proposal: Restructure tests so they are one file per model

[ekryski]

My suggestion is to split up the tests so that they mirror the folder structure similar to the source files, and have one test file per model. That way we can have filters for test runs to enable/disable certain tests (ie. smoke tests or other heavier weight tests). This way we can not only easily filter tests based on how compute intensive they are but it will also make it easier to run only specific model tests - saving time on CI and make it easier to reason about test coverage for a given model.

Additional benefits would be to improve merge ability, so that we don't have conflicts in the smoke test file (something I've run into a few times in particular) as well as smaller, more discrete files reduce the likelihood of AI hallucinations when reading/writing to the files.

In my own fork and upstream project using mlx-audio-swift I have a comprehensive set of tests and benchmarks and would love to contribute some of that upstream. I've had to do a lot of the testing outside of mlx-audio-swift so as not to pollute the repo and because the current structure is fragile and not as robust. I'm testing things like WER, latency, system memory and cpu usage, etc.

[lucasnewman]

I'm open to it -- is there a way to stage in the work?

1. Separate tests per-model (should be straightforward and mechanical)
2. Add metric reporting and benchmarks as an evaluation suite

[ekryski]

Yeah, I was thinking, two separate phases/PRs. Exactly the way you described.

[Blaizzy]

My suggestion is to split up the tests so that they mirror the folder structure similar to the source files, and have one test file per model. That way we can have filters for test runs to enable/disable certain tests (ie. smoke tests or other heavier weight tests). This way we can not only easily filter tests based on how compute intensive they are but it will also make it easier to run only specific model tests - saving time on CI and make it easier to reason about test coverage for a given model.

This can be done by:

//  Run all smoke tests (serialized):
//    xcodebuild test \
//      -scheme MLXAudio-Package \
//      -destination 'platform=macOS' \
//      -only-testing:MLXAudioTests/Smoke \
//      CODE_SIGNING_ALLOWED=NO
//
//  Run a single category:
//    -only-testing:'MLXAudioTests/Smoke/CodecsSmokeTests'
//    -only-testing:'MLXAudioTests/Smoke/TTSSmokeTests'
//    -only-testing:'MLXAudioTests/Smoke/STTSmokeTests'
//    -only-testing:'MLXAudioTests/Smoke/VADSmokeTests'
//    -only-testing:'MLXAudioTests/Smoke/STSSmokeTests'
//
//  Run a single test (note the trailing parentheses for Swift Testing):
//    -only-testing:'MLXAudioTests/Smoke/STTSmokeTests/qwen3ASRTranscribe()'
//
//  Filter test results:
//   2>&1 | grep --color=never -E '(^􀟈 |^􁁛 |^􀢄 |^\*\* TEST|\x1b\[1;35m|model loaded|Encoded to|Reconstructed audio|Generating audio|Generated audio|Generated [0-9]+ tokens|Streaming|Saved |Received final|Found [0-9]|Processing time|Streaming complete|Chunk [0-9]|  [Tt]ext:|  prompt_tokens|  generation_tokens|  total_tokens|  prompt_tps|  generation_tps|total_time| peak_memory|Peak Memory|Prompt:.*tokens/s|Prompt Tokens|Total Time|SPEAKER audio|Sortformer Output|Audio input shape|Loading.*model|Loaded audio|ForcedAligner|Running forced|\[.*s - .*s\])'

@lucasnewman we can just generate coments like this and add it to the top of all test files

[Blaizzy]

My concern with tests per model is that we will soon be looking at hundreds of files to maintain and will make GH actions to be overloaded.

[Blaizzy]

Additional benefits would be to improve merge ability, so that we don't have conflicts in the smoke test file (something I've run into a few times in particular)

hmmm, smoke tests are optional. They are my way to manually check a models generate and generateStream. Could you elaborate on the conflicts?

as well as smaller, more discrete files reduce the likelihood of AI hallucinations when reading/writing to the files.

This is upto the contributor as the "manager" of the agent.

I find it harder for the AI agents to produce proper and generalizable tests when you don't have a file with a general structure for all models. The tend to agents make up custom tests for every single model which is not maintainable.

In other words, I don't want to incentivize vibe testing and if anyone wants to do it it's much better for the model to read a single file with lots of examples of how to do it right.

[lucasnewman]

@Blaizzy I put up #94 which has the improved docs (and some important fixes) so both humans and agents can run them quickly and easily.

[Blaizzy]

Perfect, thanks!

I actually use these comments locally when I want to run a test or ask an agent to run it.

Now that's clear everywhere it should be easier.

[ekryski]

My concern with tests per model is that we will soon be looking at hundreds of files to maintain and will make GH actions to be overloaded.

Yeah, I was already doing all that. I don't necessarily see more files being an issue. In fact, it's actually better for LLMs to code against because they won't need to parse every single file every time we want to improve tests or modify them, for a particular model. But maybe that's also biased by a bit of my own personal preference... always prefer to keep files as small as possible for human (and now machine) readability.

hmmm, smoke tests are optional. They are my way to manually check a models generate and generateStream. Could you elaborate on the conflicts?

Oh, it was just more of a pain when I was building PRs for porting different models because we're working with the same smoke test files and there's almost always a merge conflict because the code for one models test would be inserted at the same line of a file as another model. Could also be a "me" issue re: git merge strategy.

I also use the smoke tests because I am running unit and integration tests to ensure the model actually does things in the upstream in my own project that uses mlx-audio-swift I have a much more robust set of tests + benchmarks.

---

At the end of the day, whatever you guys would prefer. I'm also happy to keep my tests and benches out of the repo. Been around the open source game a long time, so I know how much of a bitch it can be to manage and triage issues and PRs. Even worse, with LLM sloppy coding. 😄

[lucasnewman]

At the end of the day, whatever you guys would prefer. I'm also happy to keep my tests and benches out of the repo. Been around the open source game a long time, so I know how much of a bitch it can be to manage and triage issues and PRs. Even worse, with LLM sloppy coding. 😄

fwiw, I think it would be great to have a benchmark / eval suite!

[Blaizzy]

fwiw, I think it would be great to have a benchmark / eval suite!

I second that, I made a branch in python but its more related to accuracy.

Blaizzy/mlx-audio#490

Let me know if you have something different in mind and we can add it.

[Blaizzy]

Yeah, I was already doing all that. I don't necessarily see more files being an issue. In fact, it's actually better for LLMs to code against because they won't need to parse every single file every time we want to improve tests or modify them, for a particular model. But maybe that's also biased by a bit of my own personal preference... always prefer to keep files as small as possible for human (and now machine) readability.

At the end of the day, whatever you guys would prefer. I'm also happy to keep my tests and benches out of the repo. Been around the open source game a long time, so I know how much of a bitch it can be to manage and triage issues and PRs. Even worse, with LLM sloppy coding. 😄

I hear you, but there are a few practical reasons I'd prefer keeping tests consolidated:

• CI scale — splitting tests per model means hundreds of separate jobs per PR. At our current repo velocity, that's thousands of Actions runs per week, which gets noisy fast and makes failures harder to isolate.
• Ecosystem precedent — this is the convention in most ML project such as transformers and the MLX ecosystem.
• Uniformity — the vast majority of model tests follow the same pattern: config, inference (with kwargs), dtype cast, and output shape. Keeping them together makes that structure obvious and easy to scan.

The LLM readability point is fair, but I'd argue the opposite: a consistent, flat file is actually easier for a model and human to reason about because the patterns repeat predictably. Fragmentation across files adds navigation overhead without much structural benefit here.

Oh, it was just more of a pain when I was building PRs for porting different models because we're working with the same smoke test files and there's almost always a merge conflict because the code for one models test would be inserted at the same line of a file as another model. Could also be a "me" issue re: git merge strategy.

That's actually one of the core problems git and github solves well, two PRs adding tests to the same file at different locations should merge cleanly in most cases and when it doesn't there is a merge conflict resolution UI here and in your IDE.

Maybe I'm missing something but I personally haven't run into this issue in any of the projects I maintain. However, if it's happening consistently, it might be worth looking at the merge strategy or how the PRs are rebased. Happy to take a look at a concrete and reproducible example if you can share one. That's worth addressing properly rather than restructuring the test layout around it.