[Broken Build]: Dotnet Publish is not idempotent and deletes required Content

[EdLichtman]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

With certain configuration from prior to net8sdk, I could dotnet publish to my heart's content and everything worked.

Now, if I have a Content[xml][CopyToPublishDirectory:Always] file, then the first time I publish the project, the xml gets copied to the publish directory. The second time I publish the project, the xml is deleted from the publish directory and cannot be republished until I completely clean the directory or change the csproj file.

Expected Behavior

I expect that if I have

<ItemGroup>
    <Content Include="test.xml" CopyToPublishDirectory="Always" CopyToOutputDirectory="Always"/>
</ItemGroup>

in my project, then when I run dotnet publish, my test.xml file comes with me to the final publish location -- every single time I run it.

Steps To Reproduce

Download my source code:
https://github.com/EdLichtman/NUGET_HOME_ISSUES_12941/

Set Powershell Variable

$Csproj = "Path:\To\My\SourceCode\UsingPackagedVulnerability\UsingPackagedVulnerability.csproj

Run the following Powershell

dotnet publish $csproj --nologo --self-contained --configuration Release --framework net6.0 --runtime win-x64 --verbosity minimal

See that in the publish directory, there is a test.xml file.

Run the following Powershell

# Repeat the dotnet publish command above
dotnet publish $csproj --nologo --self-contained --configuration Release --framework net6.0 --runtime win-x64 --verbosity minimal

See that the test.xml file is gone.

Keep repeating the command, and continue to see that it is not publishing the test.xml ever.

Exceptions (if any)

No response

.NET Version

8.0.100

Anything else?

The only workaround is manually deleting the folder contents myself, or changing the csproj file.

If I delete the folder contents myself, then it always reproduces the xml file.

If I set DeleteExistingFiles:true, then it doesn't delete the folder contents before this happens and therefore doesn't solve the problem.

If I set DeleteExistingFiles:false, then it still deletes the test.xml file that I've added. I was curious to see if it was something wrong with the build order where DeleteExistingFiles was run after the test.xml was copied down. This is not the case.

The only workaround I have, which is absolutely unacceptable is to add the following to every single project I have:

  <Target Name="CleanPublishDirectoryBeforePublish" BeforeTargets="Compile">
    <ItemGroup>
      <PublishedFiles Include="$(PublishDir)\*"/>
    </ItemGroup>
    <Delete Files="@(PublishedFiles)"/>
  </Target>

It's unacceptable for a number of reasons, one of which though includes "We don't build during publish". We run dotnet build, then later on we run dotnet publish --no-restore --no-build. So if I don't run compile, then the Clean Target I created won't run during the publish anyway. I tried saying BeforeTargets="Publish", and dotnet publish still deleted the test.xml without recreating it.

[EdLichtman]

FYI I just uninstalled net8 and am using sdk 7.0.401, and when I run steps to reproduce, there is no issue. Therefore this is absolutely a breaking change from net8.0.100

[EdLichtman]

turns out I can't uninstall because it comes with our team's required install of Visual Studio. However, I figured out a "better" workaround.

I added the following to a Directory.Build.targets file in the root of my projects. By targeting before the "PrepareForBuild" step, it runs well before publish. If I target before the "Publish" step, the issue is still there, so all I know is this bug takes place somewhere between "PrepareForBuild" and "Publish".

    <Target Name="CleanPublishDirectoryBeforePublish" 
      BeforeTargets="PrepareForBuild" 
      Condition="'$(TargetFramework)' == 'net6.0' ">
      <ItemGroup>
        <PublishedFiles Include="$(PublishDir)\*"/>
      </ItemGroup>
      <Delete Files="@(PublishedFiles)"/>
    </Target>

I just wish there were a way to say "only run before PrepareForBuild, under the condition that Publish will also eventually be run"

This workaround is more acceptable, but it's not a good and proper fix.

[marcpopMSFT]

I tried this with a simple console app using the 8.0.100 SDK but the app targeting net6.0 to try to mimic your scenario as closely as possible. I do not see the test.xml file get removed. If you can collect an https://aka.ms/binlog during the publish that is removing the file, we might be able to track down what target is trying to delete that file and why it's not being copied every time.

[EdLichtman]

Well, we had to change certain things, like we can no longer dotnet build, we can only dotnet publish. If we build, then we get an error where files that were built don't get moved to the publish project. And now I can no longer reproduce this locally.

[EdLichtman]

Oh! By the way, if somehow someone else lands on this I found that there is an api to the msbuild item that I believe may have fixed this for me.

So the problem was two-fold:

1. The build was not idempotent -- for a reason I can't necessarily capture anymore.
2. When the build was re-run after it had been run the first time, a particular content file used in Log4Net was deleted

So in this case I misused the problem 1 as a solution to problem 2. Because the file was deleted after the second publish, I basically just wiped the directory before each build.

The way I ended up fixing problem 2, in case you need to fix problem 2, is by labeling the file with ExcludeFromSingleFile:

<ExcludeFromSingleFile>true</ExcludeFromSingleFile>