Description
Issue Number 1: Use buildtransitive
I have run into many runtime issues because the runtime dlls are missing. This is because librdkafka.redist doesn't use the buildtransitive folder, which allows projects that transitively consume this package to the targets file. It is very common for my dependency tree to look like librdkafka.redist <- Confluent.Kafka <- library <- application.
The first place that Confluent.Kafka looks for the native dll is {location of .dll/.exe usually bin/Debug/net48/}/librdkafka/x64/librdkafka.dll for net48 apps. See the code here. These files are copied there by build/librdkafka.redist.props. Unfortunately, since this file is is in the build folder, only projects that directly reference librdkafka.redist will get the runtime dlls copied into it.
See MS's docs on how to wire up a NuGet package here. Or, see docs on runtime libs here. Essentially, you need to copy your build folder into a buildTransitive folder. Per MS the buildTransitive folder is:
MSBuild .targets and .props files that flow transitively to any consuming project. See the feature page.
Here is what your package looks like. See https://nuget.info/packages/librdkafka.redist/2.8.0 for more info.
In summary, issue 1 is to use the buildTransitive folder.
Potential Issue Number 2: fix librdkafka.redist.targets location and ReferenceCopyLocalPaths
Potential issue number 2 is that I'm not sure your targets file is being imported. If I understand MS's docs correctly, it needs to move up one directory so that the build actually uses it. So, "build/librdkafka.redist.targets".
This creates issue 2.b, since librdkafka.redist.targets adds updates to ReferenceCopyLocalPaths. This might mean that MSBuild will copy an extra file directly in the bin. So, you'll have the library copied in 2 (technically 3) places. So, you should probably remove this or make it conditional on a certain case.
So this
$ find . -name librdkafka.dll
bin/Debug/net48/librdkafka.dll
bin/Debug/net48/librdkafka/x64/librdkafka.dll
bin/Debug/net48/librdkafka/x86/librdkafka.dll
becomes
$ find . -name librdkafka.dll
bin/Debug/net48/librdkafka/x64/librdkafka.dll
bin/Debug/net48/librdkafka/x86/librdkafka.dll
In summary, move librdkafka.redist.targets up a directory and remove ReferenceCopyLocalPaths.
Issue Number 3: make the bin librdkafka folder conditional
Related to 2.b. In .NET 5 and greater MSBuild automatically copies the dlls into the runtime folder. This means you will have dlls in bin/Debug/net8.0/librdkafka and bin/Debug/net8.0/runtimes/**. The solution is update the ItemGroup that sets up the librdkafka folder (<Content Include="$(MSBuildThisFileDirectory)..\runtimes\win-x86\native\*">) to be conditionally on the target framework is NOT >= .NET5.0. I can give you the exact MSBuild code for this if needed.
In summary, make the bin librdkafka folder conditional upon NOT (.NET 5 or greater).
Issue Number 4: Fix Package Generation Warnings
When you build "librdkafka.redist" in a package (GeneratePackageOnBuild=true), it generates a bunch of NU5118 warnings. This is because of the ItemGroup that sets up the librdkafka folder (<Content Include="$(MSBuildThisFileDirectory)..\runtimes\win-x86\native\*">). I'm not sure exactly how to fix this. One idea would be to only copy if the apps is not a library (see IsNotALibrary below). Another idea is to change the ItemGroup from a Content tag to a None tag. This might make it so that MSBuild doesn't try to keep copying them.
C:\Program Files\dotnet\sdk\9.0.103\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(221,5): warning NU5118: File 'C:\Users\mbussing\.nuget\packages\librdkafka.redist\2.8.0\runtimes\win-x64\native\libcurl.dll' is not added because the package already contains file 'content\libcurl.dll' [C:\Users\mbussing\Downloads\KafkaRuntimeRepro\KafkaRuntimeRepro.csproj]
C:\Program Files\dotnet\sdk\9.0.103\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(221,5): warning NU5118: File 'C:\Users\mbussing\.nuget\packages\librdkafka.redist\2.8.0\runtimes\win-x64\native\libcurl.dll' is not added because the package already contains file 'contentFiles\any\net8.0\libcurl.dll' [C:\Users\mbussing\Downloads\KafkaRuntimeRepro\KafkaRuntimeRepro.csproj]
C:\Program Files\dotnet\sdk\9.0.103\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(221,5): warning NU5118: File 'C:\Users\mbussing\.nuget\packages\librdkafka.redist\2.8.0\runtimes\win-x64\native\librdkafka.dll' is not added because the package already contains file 'content\librdkafka.dll' [C:\Users\mbussing\Downloads\KafkaRuntimeRepro\KafkaRuntimeRepro.csproj]
<PropertyGroup>
<IsWebsite>False</IsWebsite>
<IsWebsite Condition="'$(ProjectTypeGuids)' != '' AND $([System.Text.RegularExpressions.Regex]::IsMatch($(ProjectTypeGuids), '349c5851-65df-11da-9384-00065b846f21'))">True</IsWebsite>
<IsNotALibrary>False</IsNotALibrary>
<IsNotALibrary Condition="'$(OutputType)' != 'Library' OR '$(IsTestProject)' == 'True' OR '$(IsWebsite)' == 'True'">True</IsNotALibrary>
</PropertyGroup>
At the end of the day, you should be able to create a NuGet package consuming this without these warnings AND parent projects that consumed
librdkafka.redist transitively should not have runtime issues.
repros
You can see the output via:
<PropertyGroup>
<TargetFramework>net48</TargetFramework>
<OutputType>Exe</OutputType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Confluent.Kafka" Version="2.8.0" />
</ItemGroup>
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<OutputType>Exe</OutputType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Confluent.Kafka" Version="2.8.0" />
</ItemGroup>
For issue number 4
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Confluent.Kafka" Version="2.8.0" />
<PackageReference Include="librdkafka.redist" Version="2.8.0" />
</ItemGroup>
</Project>
For issue number 1, the buildTransitive issue, you'll need to create dep tree like: Confluent.Kafka <- library <- application.
I can repro these issues if needed.
final thoughts
Thanks for taking a look at this! Sorry there is so much. There was a lot to discuss.