CI: Android test suite is failing since January 2026

[lefou]

I haven't analyzed it and am not sure, it's always the same set of tests. Hence, I opened this issue to track it.

[lihaoyi]

@vaslabs could you help look into the failures

[vaslabs]

looks like since the emulator changes, it's picking android-35 for the image name, but there are no android-35 images (only android-34). I wonder why it didn't fail before the PR was merged, but will fix it asap

cc: @souvlakias

No I was wrong, it seems something else broke it

[vaslabs]

maybe something with parallelisation, I ran a few individually and they work, but all together are conflicting on the emulator

[vaslabs]

the third party compose samples pass, what's the difference between shared and packaged daemon?

[vaslabs]

I can replicate it locally, running with shared doesn't seem to be able to read the android device configuration

[lihaoyi]

.shared checks out consecutive runs in the same workspace folder and with the daemon living between them, .packaged runs in separate folders with a fresh daemon per test.

We can swap it to .packaged if it doesn't work with .shared I guess, but it would be nice to figure out why consecutive tests running in the same folder fail since Mill is meant to handle cleanup and invalidation automatically

[vaslabs]

.shared checks out consecutive runs in the same workspace folder and with the daemon living between them, .packaged runs in separate folders with a fresh daemon per test.

We can swap it to .packaged if it doesn't work with .shared I guess, but it would be nice to figure out why consecutive tests running in the same folder fail since Mill is meant to handle cleanup and invalidation automatically

yeah, I can patch it up for now, I suspect the android setup won't play so well as the emulator management is a bit hacky and the configuration is between global/system and task directories.

there's another issue with packaged, all emulators start more or less at the same moment and I get some OOM errors locally.

I did run 1 at a time to make sure emulators don't conflict, locally this works with

./mill -ij1 example.androidlib.__.packaged.daemon.testForked

for future reference, we need to handle the emulator in a way that works for android, android dev daily activities and CI, I'm not sure how to do all of that yet in a nice way, we need to try a few ways + let mill handle the emulator process better to start with (e.g. make the clean startEmulator work instead of having a stop emulator) so mill knows about the process

[vaslabs]

For now packaged seems to work https://github.com/vaslabs-ltd/mill/actions/runs/20780364853/job/59676844534

[lefou]

It's probably ok to not run more than one Android emulator at once. We could hard-code some synchronization, even if that means Mill is potentially wasting some job slots due to blocking.

We could also add some support into the evaluator in combination with a new concurrency Task property to manage concurrency, since we faced resource hungry task also in other places (#6226, and didn't agree on blocking with a semaphore), but that's probably a far stretch.

[vaslabs]

another quick thing for the CI would be to download the emulator images beforehand, it looks like now the sdk installation happens concurrently and the files get corrupted .

We can fix with a file lock as well or see if there's a way to isolate the download (as other things, like android studio , maybe downloading the same image at the same time)