File locked by System process in Docker build

[rainersigwald]

hi @rainersigwald following information will be helpful for you, please ping me anytime if you need further clarification.

- What kind of project(s) do you have? :

I will tell here all the background information with all the tools we using. We having a private repository in GIT calls "TownSuite/WebPortals" it is a Web Site based on ASP.NET framework and mainly C#,Java Script languages . We have several Multiple c# projects and I will explain you the exact build process later.

The GIT source code build, Test & deployment process we initiated by using our Jenkins CI server. So we have Jenkins master server connecting with Jenkins windows slave node. So Jenikins downloading "TownSuite/WebPortals" Source-Code on to that windows Slave node when start the Build process via Jenkins. Inside that Jenkins windows slave node we have installed Docker Desktop for the purpose of run Windows docker containers on top of that Windows slave node. So that downloaded Source-code we can mount inside the docker container.

- Which Docker image are you using? :

We using a Windows based Docker image. Actually we having powershell scripts (buildrelease-ng-docker.ps1 & buildrelease-ng.ps1) running within our Jenkins file for build and test our WebProjects. So first we run buildrelease-ng-docker.ps1 as below for run the docker container.

Here I highlighted in yellow the docker build image.

- How are you building it in the Docker container?

So as you can see in above image first we running the docker image and we running out build script ".\buildrelease-ng.ps1" at the same time inside the docker container. So all our build tool (MSBuild) and Build command we specifiy inside that buildrelease-ng.ps1 script. I will show an image of inside the buildrelease-ng.ps1 script as below.

So here I Highlighted only a one of the build commands for build one Web.Sln project. Here MSBuild tool we already have installed inside the docker container in following path. $MSBUILD="C:\BuildTools\MSBuild\15.0\Bin\MSBuild.exe".

- What is the exact error you're seeing, including the file that is locked?

Below is the exact I'm getting on my Jenkins Log Output when Building inside docker container by using the MSBuild tool.

This is only one error these kins of errors happening several times on the log when copying several different dll files.

- If possible, please include binary logs ?

I don't know how to get binary logs. Is it the logs for MsBuild ?

Originally posted by @AmilaDevops in #4743 (comment)

[rainersigwald]

Thanks for the details, @AmilaDevops. It's very interesting that the process System is what's locking the files. I don't know exactly how Windows containers work; in Linux containers PID 1 is the entry-point process which may not be init; if that's the case here it might be misreporting PID 4 as System instead of . . . the PowerShell host? Or some other process?

Does your failure happen in the first call to MSBuild within the script, or a subsequent one? Always the same place?

Does your build create any task assemblies? If so, are they used within the same MSBuild invocation, in a subsequent one, or both?

Can you change your build script to log all processes with PIDs and command lines before the failing MSBuild invocation? That could help identify PID 4 more clearly.

I don't know how to get binary logs. Is it the logs for MsBuild ?

https://aka.ms/msbuild/binlog has instructions (and warnings about what is in the logs--make sure you're willing to share them).

[AmilaDevops]

No actually there were different dll files when copying gives this error such as once the WebDbMigrations.dll and once the Townsuite.Web.Poco.dll (like in above screen-shot) sometimes another dll file.

Yes that happenes when the first call to MSBuild within the buildrelease-ng.ps1 script. Such as when calling MSBuild in below highlighted when our first building project name is "Web.sln" ,

When building that project "Web.sln" its normally building several dll files such as WebDbMigrations.dll and Townsuite.Web.Poco.dll. So after build those dll files and when using again those dll files for copy or something (for another purpose) its generating this System error always.

Give me few hours @rainersigwald I will provide binary log files with more answers for your questions in detal. Thanks.

[AmilaDevops]

hi @rainersigwald sorry for delay in reply mate;

@rainersigwald 03 binary log files attached here respectively for 03 Web.sln projects (seperately) building by using MSBUILD as in below code. Please unzip files before use. Thanks @rainersigwald

msbuild.zip

msbuild2.zip

msbuild3.zip

[AmilaDevops]

@rainersigwald hi how are you? is there anything you can find in these attached msbuild binary log files ?

Dind't here from you....

[AmilaDevops]

hey Mr @rainersigwald, @livarcocc , @benvillalobos , @cdmihai how are you ? I know you busy but If you could help with this issue I'm really thankful, I put MsBuild binary logs files attached in above comment up.

Could You check above binary logs and let me know; Because most people cannot build their daily projects because of this Process lock issue. So if you could help with this "File locked by System process (4)" issue that will be helpful.

Thanks,

[alohaninja]

@AmilaDevops - I noticed this started happening for me once I applied Group Policy - suspect Windows Defender (MsMpEng.exe/MsSense.exe) or other system scanner is interfering with the directory. Using Win10 v2004 Docker Host - was working perfectly prior to gpupdate.

We are using locally mounted Windows Volumes for persistent storage.

MSBUILD Blocked by

##[error]C:\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(4187,5): Error MSB3026: Could not copy "obj\Debug\net472\xxxx.xxxx.dll" to "bin\Debug\net472\xxxx.xxxx.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'bin\Debug\net472\xxxx.xxxx.dll' because it is being used by another process. The file is locked by: "System (4)"

Current Approach

• Apply GPO Policy Exclusions for Persistent Storage Volume
• Apply GPO Policy Exclusion for HyperV Processes

Future ideas

• Migrate to Windows Server host instead of Win10
• Migrate to Azure Container Instances

[Thoorium]

Recently I have also started to experience this issue.

Windows 10 Pro 2004 19041.508, Docker For Windows 2.4.0.0, Windows Container.

During my build, some DLL files, not always the same ones, will start to get locked by System (4). If I try to delete these files from the docker host, it says that wmwp.exe (Hyper-V) is using the file. I've added my mounted directory to the exclusion list of Defender and the hyper-v processes too but it didn't work. Turning off Real-Time protection didn't change anything so I'm not sure if Defender is the real culprit here. Any idea?

Edit: I should also note that my Windows Container image is based on the 2019 LTSC image. The image was built a few months ago and worked fine until recently.

[dannoe]

I got the same problem, but I found a fix/workaround:

I tried to compile my app inside a docker mounted folder (via -v local:remote). I copied the code to a local folder and run it again. It didn't lock any files inside the docker container.

[yavuzekmen]

I have exactly the same issue, the only way to fix the issue is that move obj (intermediaire folder) and output folder to the container.
The issue is mainly caused when we tried to compile C# projects.

[patelriki13]

@Thoorium have you found any solution? If so, could you please share it? I am facing the same issue.

[patelriki13]

@rainersigwald any fix or solution?