`Text file busy (os error 26)` due to race condition in SVM when doing multiple installations

[frolic]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

nightly

What command(s) is the bug in?

forge build

Operating System

Linux

Describe the bug

Our monorepo has a bunch of packages, each using foundry nightly in CI (via foundry-rs/foundry-toolchain@v1). Each of these use forge build in their build step and we recently tried to turn on parallel builds (via pnpm), but it failed with

Text file busy (os error 26)

The logs weren't specific about where this was coming from, but I later saw a log line that mentioned solc-0.8.13:

Unclear if this is a foundry issue or solc issue, but it originates from forge build, so I'm starting here.

Ultimately would like to be able to run forge build for multiple foundry projects in parallel. Is this an issue with some sort of global cache? Or strictly just a ulimit thing?

[mattsse]

this is likely just a flake

this usually happens when a binary is in use and another process wants to write to the binary's location (e.g. race condition of two separate forge build processes that both try to install/run it)

can you try:
delete the installed 0.8.13

install it once: run forge build once, after which it should be installed
alternatively install with svm directly:
https://github.com/roynalnaruto/svm-rs/tree/master

[frolic]

To clarify, we only see this in CI (Github actions). Foundry is installed once at the start of the action via the official action (https://github.com/foundry-rs/foundry-toolchain) and our version is configured to nightly.

And then the action's build step runs forge build multiple times in parallel (via pnpm but soon nx, once per project/package in the monorepo).

This doesn't happen if we run the CI build step without parallelization.

[mattsse]

I see, then this is indeed the above-mentioned case.

looks like the svm file lock is not working properly, will try to investigate.

are all of these build tasks targeting the same profile?

can you perhaps add a single step to the CI that runs forge build once (this should install missing solc)

[frolic]

I'm not following. Are you saying that forge build lazily installs solc (and isn't part of the foundryup step) and because I am running this in parallel, it's trying to install solc in parallel, which is causing conflicts? And the potential resolution is to foundryup then forge build once, then running all the forge build steps in parallel should work?

[mattsse]

And the potential resolution is to foundryup then forge build once, then running all the forge build steps in parallel should work

yes, missing solc bins are lazily installed during forge build, looks like the file lock in svm that should prevent these cases where two processes potentially write to and execute the same binary isn't working as expected

This should be bypassed if you trigger forge b once before starting the parallel job

Text file busy (os error 26)

https://stackoverflow.com/a/16764967/6242846

[frolic]

Will give that a try, thank you!

Aside from forge build, are there other ways to trigger getting the solc bin?

[mattsse]

forge build internally uses
https://github.com/roynalnaruto/svm-rs/tree/master
it doesn't have a static binary, please consider opening an issue there, should be possible to release it as such

[frolic]

ahh, so an alternative would be something like cargo install svm-rs && svm install 0.8.13 ahead of the parallel forge build usages? that seems a little easier to wire up in the way we have our CI actions configured

[mattsse]

yes, that works. and is essentially what forge b would do the first run

cargo install svm-rs adds a bit to CI time though, but not too much, I'd assume ~1:30min

[mattsse]

since we already have is in forge, seems like there is actually a use case for an install solc subcommand

[frolic]

since we already have is in forge, seems like there is actually a use case for an install solc subcommand

that would be great!

even nicer would be able to specify a glob or list of foundry.toml files (i.e. for monorepos) that would install the specified solc versions in them all, but I can probably script this with forge config --json

[frolic]

cargo install svm-rs adds a bit to CI time though, but not too much, I'd assume ~1:30min

I assume we could cache this so hopefully won't need this every CI run. I'm also doing this after foundryup step, so hopefully even less time needed.