CI: reduce qemu pipeline time downloading IGVM file built in the rust pipeline

[stefano-garzarella]

In .github/workflows/qemu.yml we have this TODO that we should address to reduce our CI time usage:

      ## TODO: We can replace the SVSM build step and download the IGVM file as an artifact
        # from the "Rust check" jobs instread to speed things up.

• check If it's possible to download artifacts from other pipelines related to the same run
• skip installing rust and TPM build deps, and build bin/coconut-test-qemu.igvm in qemu pipeline

CC @osteffenrh

[OshiroYuuki]

Hello! I'm relatively new to contributing to large opensource projects, so please forgive me if there is a historical reason for this setup, but I wanted to ask for some clarification on the CI pipelines.

Are we optimizing qemu.yml and rust.yml to run in parallel for speed or in series to save compute?

If we want them to run in parallel to save time, combining both into a single workflow file would solve the current bottleneck. I tried setting up a cross-workflow artifact download to pass the .igvm file to QEMU, but GitHub’s V4 Artifact architecture actively blocks parallel workflows from sharing files mid-run.

By combining the pipelines into one file, GitHub will still parallelize the tasks as separate jobs, but they will share the same Run ID and security sandbox. This lets us build Rust and configure QEMU concurrently, and instantly share the .igvm file natively without any API restrictions.

Please let me know your thoughts! I'm happy to share my work if this makes sense, but I wanted to make sure I wasn't overlooking a deliberate design choice first.

[stefano-garzarella]

Hello! I'm relatively new to contributing to large opensource projects, so please forgive me if there is a historical reason for this setup, but I wanted to ask for some clarification on the CI pipelines.

Are we optimizing qemu.yml and rust.yml to run in parallel for speed or in series to save compute?

In series because in qemu.yml we would like to re-use artifacts built by rust.yml. Currently, we are re-building them in qemu.yml.

If we want them to run in parallel to save time, combining both into a single workflow file would solve the current bottleneck. I tried setting up a cross-workflow artifact download to pass the .igvm file to QEMU, but GitHub’s V4 Artifact architecture actively blocks parallel workflows from sharing files mid-run.

They are already running in parallel because they are 2 different pipelines, no?

By combining the pipelines into one file, GitHub will still parallelize the tasks as separate jobs, but they will share the same Run ID and security sandbox. This lets us build Rust and configure QEMU concurrently, and instantly share the .igvm file natively without any API restrictions.

I'm a bit confused by parallelism, having them in 2 different pipelines is event better about parallelism IIUC. But maybe I'm missing something.

Please let me know your thoughts! I'm happy to share my work if this makes sense, but I wanted to make sure I wasn't overlooking a deliberate design choice first.

I can understand if merging the 2 pipelines is justified by the re-use of the artifacts, but I'm confused when you talk about parallelism.
If possible, I'd like to keep them in 2 different pipelines, but if this complicates too much, I'm open to merge them.

[osteffenrh]

If we want them to run in parallel to save time, combining both into a single workflow file would solve the current bottleneck. I tried setting up a cross-workflow artifact download to pass the .igvm file to QEMU, but GitHub’s V4 Artifact architecture actively blocks parallel workflows from sharing files mid-run.

Maybe I don't know enough about GitHub workflows, but right now all jobs in the same yaml run sequentially, for example "Install dependencies" then "Build".

The QEMU related tests were placed in a separate file to introduce some structure. We might have many more tests in the future, for other hypervisors and environments. IMO we should try to keep some structure, on the file/code level as well as the representation in the GitHub view.

But I agree, we could save almost the entire time the QEMU tests pipeline takes. It uses ~2 minutes for setup (even with cached QEMU) and then spends ~5 seconds to run the tests.

Since most time is spent downloading and compiling external Rust crates, maybe a cache on cargo could be an alternative that let's us keep the separate pipelines: https://github.com/marketplace/actions/rust-cache (also applies to the main rust pipeline).

[OshiroYuuki]

I sincerely apologize for my poor wording. I wasnt mentioning parallelization execution assuming the qemu workflow uses the rust igvm file. you are right at the current moment they run in parallel I worded it wrongly. I was more talking that if you wanted to have the qemu file run the rust .igvm file it would be very difficult in parallel if they were seprate workflows. If the main goal was to reduce the computational waste without combination it would be best to have the qemu file to wait for the rust workflow to finish. Im out right now i just wanted to apologize and make that quick clarification. Ill reply with a better response and explanation of my idea as soon as I can. Thank you for taking you time to reply and again sorry for my poor wording and explanation.

[OshiroYuuki]

Maybe I don't know enough about GitHub workflows, but right now all jobs in the same yaml run sequentially, for example "Install dependencies" then "Build".

No you are right I apologize I worded my comment wrong.

The QEMU related tests were placed in a separate file to introduce some structure. We might have many more tests in the future, for other hypervisors and environments. IMO we should try to keep some structure, on the file/code level as well as the representation in the GitHub view.

I totally understand that so i also thought of the idea of splitting the two workflows in to three by having a smaller program make the igvm file first then having the rust and qemu workflow use that. Im out right now ill be back soon and give a better explaintion since I totally understand the need for structure in a large open source project. Additionally though the workflow jobs and seprated and labeled on the actions tab when it runs so combining wouldnt necessarily destory structure.

But I agree, we could save almost the entire time the QEMU tests pipeline takes. It uses ~2 minutes for setup (even with cached QEMU) and then spends ~5 seconds to run the tests.

Yeah I saw that honestly i was reveiwing alot of the workflow execution logs and i honestly think the place that is really slowing down is clippy. I was thinking if we could parallelize the execution of clippy which is definitely possible but potentially tricky, you could save alot more time.

Again Ill be home soon and post a more indepth discptions as well as my code. Ill even show some other ideas i had to increase CI time since I found many ways to optimize the execution of the workflows and I would love to hear what you think. I was even working on some of the other optimizations I thought of.

[stefano-garzarella]

@OshiroYuuki no need to apologize at all, I was just confused, thanks for clarify!

Feel free to open a draft PR if you think it's easy to discuss with code. It's perfectly fine.
Thanks for your interest on this :-)