.NET 8 added as a target to all projects and tests and NJsonSchema 11 support (#2314)

* .NET 8 added as a target to all projects and tests
except examples that are built with the oldest supported version.
Semaphore is using latest version for integration tests and both version for unit tests.

* Targets netstandard2.0 on libraries for greater compatibility:
- .net 2.0+
- .NET Framework 4.6.1+
- Mono 5.4+
- Xamarin, Unity
  Newer projects such as Rules and Encryption are excluded

* NJonSchema conditional upgrade for .NET 8.0
* Run unit tests with both .NET 6 and 8
* macOS runs only .NET 8 unit tests
* Changelog entry

Author: emasab

.semaphore/semaphore.yml

@@ -2,7 +2,7 @@ version: v1.0
 name: 'confluent-kafka-dotnet build pipeline'
 agent:
   machine:
-    type: s1-prod-ubuntu20-04-amd64-1
+    type: s1-prod-ubuntu22-04-amd64-1
 global_job_config:
   prologue:
     commands:
@@ -19,10 +19,15 @@ blocks:
     task:
       agent:
         machine:
-          type: s1-prod-ubuntu20-04-amd64-2
+          type: s1-prod-ubuntu22-04-amd64-2
       jobs:
         - name: 'Build and test'
           commands:
+            # TODO: install .NET 8.0 SDK in this agent image
+            - sudo apt remove dotnet-sdk* dotnet-host* dotnet* aspnetcore* netstandard* -y
+            # TODO: remove incorrect focal repos from this agent
+            - sudo rm /etc/apt/sources.list.d/*microsoft*
+            - sudo apt update && sudo apt install -y dotnet-sdk-6.0 dotnet-sdk-8.0
             - dotnet restore
             - make build
             - make test
@@ -36,9 +41,11 @@ blocks:
         - name: 'Build and test'
           commands:
             - ulimit -n 1024
-            - dotnet restore
-            - make build
-            - make test
+            # TODO: install .NET 8.0 SDK in this agent image
+            - sudo chown -R "$(whoami)" "$(brew --caskroom)"
+            - brew install --appdir=~/Applications dotnet-sdk
+            - dotnet restore -p:TargetFramework=net8.0
+            - make test-latest
   - name: 'Windows x64'
     dependencies: [ ]
     task:
@@ -49,12 +56,12 @@ blocks:
         - name: 'Build and test'
           commands:
             - wget https://dot.net/v1/dotnet-install.ps1 -OutFile dotnet-install.ps1
-            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Version 6.0.403 -InstallDir C:\dotnet
+            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Channel 8.0 -Quality GA -InstallDir C:\dotnet
             - $Env:Path += ";C:\dotnet"
             - dotnet restore
-            - dotnet test -c ${CONFIGURATION} --no-build test/Confluent.Kafka.UnitTests/Confluent.Kafka.UnitTests.csproj
-            - dotnet test -c ${CONFIGURATION} --no-build test/Confluent.SchemaRegistry.UnitTests/Confluent.SchemaRegistry.UnitTests.csproj
-            - dotnet test -c ${CONFIGURATION} --no-build test/Confluent.SchemaRegistry.Serdes.UnitTests/Confluent.SchemaRegistry.Serdes.UnitTests.csproj
+            - dotnet test -f net8.0 -c ${CONFIGURATION} --no-build test/Confluent.Kafka.UnitTests/Confluent.Kafka.UnitTests.csproj
+            - dotnet test -f net8.0 -c ${CONFIGURATION} --no-build test/Confluent.SchemaRegistry.UnitTests/Confluent.SchemaRegistry.UnitTests.csproj
+            - dotnet test -f net8.0 -c ${CONFIGURATION} --no-build test/Confluent.SchemaRegistry.Serdes.UnitTests/Confluent.SchemaRegistry.Serdes.UnitTests.csproj
   - name: 'Windows Artifacts on untagged commits'
     run:
       when: "tag !~ '.*'"
@@ -68,7 +75,8 @@ blocks:
         - name: 'Build and push artifacts'
           commands:
             - wget https://dot.net/v1/dotnet-install.ps1 -OutFile dotnet-install.ps1
-            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Version 6.0.403 -InstallDir C:\dotnet
+            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Channel 6.0 -Quality GA -InstallDir C:\dotnet
+            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Channel 8.0 -Quality GA -InstallDir C:\dotnet
             - $Env:Path += ";C:\dotnet"
             - dotnet restore
             - dotnet build Confluent.Kafka.sln -c ${Env:CONFIGURATION}
@@ -97,7 +105,8 @@ blocks:
         - name: 'Build and push artifacts'
           commands:
             - wget https://dot.net/v1/dotnet-install.ps1 -OutFile dotnet-install.ps1
-            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Version 6.0.403 -InstallDir C:\dotnet
+            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Channel 6.0 -Quality GA -InstallDir C:\dotnet
+            - powershell -ExecutionPolicy ByPass -File dotnet-install.ps1 -Channel 8.0 -Quality GA -InstallDir C:\dotnet
             - $Env:Path += ";C:\dotnet"
             - dotnet restore
             - dotnet build Confluent.Kafka.sln -c ${Env:CONFIGURATION}
@@ -113,49 +122,43 @@ blocks:
             - dotnet pack src/Confluent.SchemaRegistry.Serdes.Protobuf/Confluent.SchemaRegistry.Serdes.Protobuf.csproj -c ${Env:CONFIGURATION} --output artifacts
             - dotnet pack src/Confluent.SchemaRegistry.Serdes.Json/Confluent.SchemaRegistry.Serdes.Json.csproj -c ${Env:CONFIGURATION} --output artifacts
             - artifact push workflow artifacts
-  - name: 'Integration tests'
+  - name: 'Integration tests and documentation'
     dependencies: [ ]
     task:
       agent:
         machine:
-          type: s1-prod-ubuntu20-04-amd64-2
+          type: s1-prod-ubuntu22-04-amd64-2
       prologue:
         commands:
           - '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY'
+          # TODO: install .NET 8.0 SDK in this agent image
+          - sudo apt remove dotnet-sdk* dotnet-host* dotnet* aspnetcore* netstandard* -y
+          # TODO: remove incorrect focal repos from this agent
+          - sudo rm /etc/apt/sources.list.d/*microsoft*
+          - sudo apt update && sudo apt install -y dotnet-sdk-8.0
       jobs:
         - name: 'Build documentation'
           commands:
-            - dotnet tool update -g docfx
-            - docfx doc/docfx.json
+            - dotnet tool update --tool-path $PWD docfx
+            - ./docfx doc/docfx.json
         - name: 'Build and test with "classic" protocol'
           commands:
             - cd test/docker && docker-compose up -d && sleep 30 && cd ../..
             - export SEMAPHORE_SKIP_FLAKY_TESTS='true'
-            - dotnet restore
-            - cd test/Confluent.Kafka.IntegrationTests && dotnet test -l "console;verbosity=normal" && cd ../..
+            - dotnet restore -p:TargetFramework=net8.0
+            - cd test/Confluent.Kafka.IntegrationTests && dotnet test -f net8.0 -l "console;verbosity=normal" && cd ../..
         - name: 'Build and test with "consumer" protocol'
           commands:
             - cd test/docker && docker-compose -f docker-compose-kraft.yaml up -d && cd ../..
             - sleep 300
             - export SEMAPHORE_SKIP_FLAKY_TESTS='true'
             - export TEST_CONSUMER_GROUP_PROTOCOL=consumer
-            - dotnet restore
-            - cd test/Confluent.Kafka.IntegrationTests && dotnet test -l "console;verbosity=normal" && cd ../..
-
-  - name: 'Schema registry and serdes integration tests'
-    dependencies: [ ]
-    task:
-      agent:
-        machine:
-          type: s1-prod-ubuntu20-04-amd64-2
-      prologue:
-        commands:
-          - '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY'
-      jobs:
-        - name: 'Build and test'
+            - dotnet restore -p:TargetFramework=net8.0
+            - cd test/Confluent.Kafka.IntegrationTests && dotnet test  -f net8.0  -l "console;verbosity=normal" && cd ../..
+        - name: 'Schema registry and serdes integration tests'
           commands:
             - cd test/docker && docker-compose up -d && cd ../..
             - export SEMAPHORE_SKIP_FLAKY_TESTS='true'
-            - dotnet restore
-            - cd test/Confluent.SchemaRegistry.Serdes.IntegrationTests && dotnet test -l "console;verbosity=normal" && cd ../..
-            # - cd test/Confluent.SchemaRegistry.IntegrationTests && dotnet test -l "console;verbosity=normal" && cd ../..
+            - dotnet restore -p:TargetFramework=net8.0
+            - cd test/Confluent.SchemaRegistry.Serdes.IntegrationTests && dotnet test -f net8.0 -l "console;verbosity=normal" && cd ../..
+            - cd test/Confluent.SchemaRegistry.IntegrationTests && dotnet test -f net8.0 -l "console;verbosity=normal" && cd ../..

CHANGELOG.md

@@ -5,6 +5,7 @@
 * References librdkafka.redist 2.6.0. Refer to the [librdkafka v2.6.0 release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.6.0) for more information.
 * [KIP-848 EA](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol): Admin API for listing consumer groups now has an optional filter to return only groups of given types (#2323).
 * [KIP-460](https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC) Admin Leader Election RPC (#2320)
+* .NET 8 support with NJsonSchema 11 (#2314)
 
 
 # 2.5.3

Makefile

@@ -10,7 +10,8 @@ EXAMPLE_DIRS=$(shell find ./examples -name '*.csproj' -exec dirname {} \;)
 TEST_DIRS=$(shell find ./test -name '*.csproj'   \;)
 UNIT_TEST_DIRS=$(shell find . -type d -regex '.*UnitTests$$' -exec basename {} \;)
 
-DEFAULT_FRAMEWORK?=net6.0
+# We want to run tests by default with latest version of .NET
+DEFAULT_TEST_FRAMEWORK?=net8.0
 
 all:
 	@echo "Usage:   make <dotnet-command>"
@@ -19,10 +20,15 @@ all:
 .PHONY: test
 
 build:
-	for d in $(EXAMPLE_DIRS) ; do dotnet $@ -f $(DEFAULT_FRAMEWORK) $$d; done ; \
-	for d in $(TEST_DIRS) ; do dotnet $@ -f $(DEFAULT_FRAMEWORK) $$d; done ;
+	for d in $(EXAMPLE_DIRS) ; do dotnet $@ $$d; done ; \
+	for d in $(TEST_DIRS) ; do dotnet $@ $$d; done ;
 
 test:
 	@(for d in $(UNIT_TEST_DIRS) ; do \
-		dotnet test -f $(DEFAULT_FRAMEWORK) test/$$d/$$d.csproj ; \
+		dotnet test test/$$d/$$d.csproj ; \
+	done)
+
+test-latest:
+	@(for d in $(UNIT_TEST_DIRS) ; do \
+		dotnet test -f $(DEFAULT_TEST_FRAMEWORK) test/$$d/$$d.csproj ; \
 	done)

README.md

@@ -34,17 +34,17 @@ client. Thanks Andreas!
 
 confluent-kafka-dotnet is distributed via NuGet. We provide the following  packages:
 
-- [Confluent.Kafka](https://www.nuget.org/packages/Confluent.Kafka/) *[net462, netstandard1.3, netstandard2.0]* - The core client library.
-- [Confluent.SchemaRegistry.Serdes.Avro](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Avro/) *[netstandard2.0]* - Provides a serializer and deserializer for working with Avro serialized data with Confluent Schema Registry integration.
-- [Confluent.SchemaRegistry.Serdes.Protobuf](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Protobuf/) *[netstandard2.0]* - Provides a serializer and deserializer for working with Protobuf serialized data with Confluent Schema Registry integration.
-- [Confluent.SchemaRegistry.Serdes.Json](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Json/) *[netstandard2.0]* - Provides a serializer and deserializer for working with Json serialized data with Confluent Schema Registry integration.
-- [Confluent.SchemaRegistry](https://www.nuget.org/packages/Confluent.SchemaRegistry/) *[netstandard2.0]* - Confluent Schema Registry client (a dependency of the Confluent.SchemaRegistry.Serdes packages).
-- [Confluent.SchemaRegistry.Encryption](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption/) *[netcoreapp3.1, net6.0]* - Confluent Schema Registry client-side field-level encryption client (a dependency of the other Confluent.SchemaRegistry.Encryption.* packages).
-- [Confluent.SchemaRegistry.Encryption.Aws](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Aws/) *[netcoreapp3.1, net6.0]* - Confluent Schema Registry client-side field-level encryption client for AWS KMS.
-- [Confluent.SchemaRegistry.Encryption.Azure](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Azure/) *[netcoreapp3.1, net6.0]* - Confluent Schema Registry client-side field-level encryption client for Azure Key Vault.
-- [Confluent.SchemaRegistry.Encryption.Gcp](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Gcp/) *[netcoreapp3.1, net6.0]* - Confluent Schema Registry client-side field-level encryption client for Google Cloud KMS.
-- [Confluent.SchemaRegistry.Encryption.HcVault](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.HcVault/) *[netcoreapp3.1, net6.0]* - Confluent Schema Registry client-side field-level encryption client for Hashicorp Vault.
-- [Confluent.SchemaRegistry.Rules](https://www.nuget.org/packages/Confluent.SchemaRegistry.Rules/) *[net6.0]* - Confluent Schema Registry client-side support for data quality rules (via the Common Expression Language) and schema migration rules (via JSONata).
+- [Confluent.Kafka](https://www.nuget.org/packages/Confluent.Kafka/) *[netstandard2.0, net6.0, net8.0]* - The core client library.
+- [Confluent.SchemaRegistry.Serdes.Avro](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Avro/) *[netstandard2.0, net6.0, net8.0]* - Provides a serializer and deserializer for working with Avro serialized data with Confluent Schema Registry integration.
+- [Confluent.SchemaRegistry.Serdes.Protobuf](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Protobuf/) *[netstandard2.0, net6.0, net8.0]* - Provides a serializer and deserializer for working with Protobuf serialized data with Confluent Schema Registry integration.
+- [Confluent.SchemaRegistry.Serdes.Json](https://www.nuget.org/packages/Confluent.SchemaRegistry.Serdes.Json/) *[netstandard2.0, net6.0, net8.0]* - Provides a serializer and deserializer for working with Json serialized data with Confluent Schema Registry integration.
+- [Confluent.SchemaRegistry](https://www.nuget.org/packages/Confluent.SchemaRegistry/) *[netstandard2.0, net6.0, net8.0]* - Confluent Schema Registry client (a dependency of the Confluent.SchemaRegistry.Serdes packages).
+- [Confluent.SchemaRegistry.Encryption](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption/) *[net6.0, net8.0]* - Confluent Schema Registry client-side field-level encryption client (a dependency of the other Confluent.SchemaRegistry.Encryption.* packages).
+- [Confluent.SchemaRegistry.Encryption.Aws](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Aws/) *[net6.0, net8.0]* - Confluent Schema Registry client-side field-level encryption client for AWS KMS.
+- [Confluent.SchemaRegistry.Encryption.Azure](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Azure/) *[net6.0, net8.0]* - Confluent Schema Registry client-side field-level encryption client for Azure Key Vault.
+- [Confluent.SchemaRegistry.Encryption.Gcp](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.Gcp/) *[net6.0, net8.0]* - Confluent Schema Registry client-side field-level encryption client for Google Cloud KMS.
+- [Confluent.SchemaRegistry.Encryption.HcVault](https://www.nuget.org/packages/Confluent.SchemaRegistry.Encryption.HcVault/) *[net6.0, net8.0]* - Confluent Schema Registry client-side field-level encryption client for Hashicorp Vault.
+- [Confluent.SchemaRegistry.Rules](https://www.nuget.org/packages/Confluent.SchemaRegistry.Rules/) *[net6.0, net8.0]* - Confluent Schema Registry client-side support for data quality rules (via the Common Expression Language) and schema migration rules (via JSONata).
 
 To install Confluent.Kafka from within Visual Studio, search for Confluent.Kafka in the NuGet Package Manager UI, or run the following command in the Package Manager Console:
 

examples/Web/Web.csproj

@@ -5,7 +5,8 @@
   </PropertyGroup>
 
   <ItemGroup>
-    <PackageReference Include="Confluent.Kafka" Version="2.6.0" />
+    <!-- nuget package reference: <PackageReference Include="Confluent.Kafka" Version="2.6.0" /> -->
+    <ProjectReference Include="../../src/Confluent.Kafka/Confluent.Kafka.csproj" />
   </ItemGroup>
 
 </Project>

src/ConfigGen/ConfigGen.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFramework>net6.0</TargetFramework>
+    <TargetFrameworks>net6.0;net8.0</TargetFrameworks>
     <LangVersion>7.1</LangVersion>
   </PropertyGroup>