Azure Split - Step 1 - Prep Work (dotnet#3902)

Author: paulmedynski

.editorconfig

@@ -162,7 +162,7 @@ dotnet_diagnostic.xUnit1031.severity=none
 dotnet_diagnostic.xUnit1030.severity=none
 
 # Xml files
-[*.{xml,csproj,stylecop,resx,ruleset,props,targets,config,nuspec}]
+[*.{xml,slnx,proj,csproj,stylecop,resx,ruleset,props,targets,config,nuspec}]
 indent_size = 2
 
 # Shell scripts

.github/workflows/codeql.yml

@@ -13,9 +13,9 @@ name: "CodeQL Advanced"
 
 on:
   push:
-    branches: [ "main" ]
+    branches: [ "main", "feat/*" ]
   pull_request:
-    branches: [ "main" ]
+    branches: [ "main", "feat/*" ]
   schedule:
     - cron: '33 23 * * 6'
 
@@ -84,9 +84,22 @@ jobs:
     - name: Run manual build steps
       if: matrix.build-mode == 'manual'
       shell: bash
+      # TODO(https://sqlclientdrivers.visualstudio.com/ADO.Net/_workitems/edit/41729):
+      # For some reason, the StressTests projects fail to restore in this
+      # environment, so we skip them by explicitly building all of the other
+      # projects instead of the main solution file.
       run: |
-        mkdir packages
-        dotnet build src/
+        dotnet build src/Microsoft.SqlServer.Server
+        dotnet build src/Microsoft.Data.SqlClient/netcore/ref
+        dotnet build src/Microsoft.Data.SqlClient/netcore/src
+        dotnet build src/Microsoft.Data.SqlClient/netfx/ref
+        dotnet build src/Microsoft.Data.SqlClient/netfx/src
+        dotnet build src/Microsoft.Data.SqlClient/src
+        dotnet build src/Microsoft.Data.SqlClient/add-ons/AzureKeyVaultProvider
+        dotnet build src/Microsoft.Data.SqlClient/tests/UnitTests
+        dotnet build src/Microsoft.Data.SqlClient/tests/FunctionalTests
+        dotnet build src/Microsoft.Data.SqlClient/tests/ManualTests
+        dotnet build src/Microsoft.Data.SqlClient/tests/PerformanceTests
 
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@v4

BUILDGUIDE.md

@@ -25,6 +25,8 @@ Once the environment is setup properly, execute the desired set of commands belo
 
 ### Targets
 
+The following build targets are defined in `build.proj`:
+
 |Target|Description|
 |-|-|
 |`BuildAllConfigurations`|Default target. Builds the .NET Framework and .NET drivers for all target frameworks and operating systems.|
@@ -33,33 +35,34 @@ Once the environment is setup properly, execute the desired set of commands belo
 |`BuildNetFx`|Builds the .NET Framework driver for all target frameworks.|
 |`BuildTestsNetCore`|Builds tests for the .NET driver.|
 |`BuildTestsNetFx`|Builds tests for the .NET Framework driver.|
-|`Clean`|Cleans generated files.|
-|`Restore`|Restores Nuget packages.|
+|`Clean`|Cleans all generated files.|
+|`Restore`|Restores NuGet packages.|
 |`RunTests`|Runs the unit, functional, and manual tests for the .NET Framework and .NET drivers|
 |`RunUnitTests`|Runs just the unit tests for the .NET Framework and .NET drivers|
 |`RunFunctionalTests`|Runs just the functional tests for the .NET Framework and .NET drivers|
 |`RunManualTests`|Runs just the manual tests for the .NET Framework and .NET drivers|
 |`BuildAkv`|Builds the Azure Key Vault Provider package for all supported platforms.|
 
-
 ### Parameters
+
+The following parameters may be defined as MSBuild properties to configure the
+build:
+
 |Name|Supported Values|Default|Description|
 |-|-|-|-|
 |`Configuration`|`Debug`, `Release`|`Debug`|Sets the release configuration.|
-|`BuildNetFx`|`true`, `false`|`true` (Windows), `false` (other)|If false, skips building the .NET Framework driver on Windows.|
 |`OSGroup`|`Unix`, `Windows_NT`, `AnyOS`|typically defaults to the client system's OS, unless using `BuildAllConfigurations` or an `AnyOS` specific target|The operating system to target.|
 |`Platform`|`AnyCPU`, `x86`, `x64`, `ARM`, `ARM64`|`AnyCPU`|May only be set when using package reference type or running tests.|
 |`TestSet`|`1`, `2`, `3`, `AE`|all|Build or run a subset of the manual tests. Omit (default) to target all tests.|
 |`DotnetPath`|Absolute file path to an installed `dotnet` version.|The system default specified by the path variable|Set to run tests using a specific dotnet version (e.g. C:\net6-win-x86\)|
 |`TF`|`net8.0`, `net462`, `net47`, `net471`, `net472`, `net48`, `net481`|`net8.0` in netcore, `net462` in netfx|Sets the target framework when building or running tests. Not applicable when building the drivers.|
 |`ResultsDirectory`|An absolute file path|./TestResults relative to current directory|Specifies where to write test results.|
 
-
 ## Example Workflows using MSBuild (Recommended)
+
 Using the default configuration and running all tests:
 
 ```bash
-msbuild
 msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetCore
 msbuild -t:RunTests
@@ -68,28 +71,25 @@ msbuild -t:RunTests
 Using the Release configuration:
 
 ```bash
-msbuild -p:configuration=Release
-msbuild -t:BuildTestsNetFx -p:TF=net462 -p:configuration=Release
-msbuild -t:BuildTestsNetCore -p:configuration=Release
-msbuild -t:RunTests -p:configuration=Release
+msbuild -t:BuildTestsNetFx -p:TF=net462 -p:Configuration=Release
+msbuild -t:BuildTestsNetCore -p:Configuration=Release
+msbuild -t:RunTests -p:Configuration=Release
 ```
 
 Running only the unit tests:
 
 ```bash
-msbuild
 msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetCore
 msbuild -t:RunUnitTests
 ```
 
-Using a specific dotnet version/architecture:
+Using a specific .NET runtime to run tests:
 
 ```bash
-msbuild -p:configuration=Release
-msbuild -t:BuildTestsNetFx -p:TF=net462 -p:configuration=Release
-msbuild -t:BuildTestsNetCore -p:configuration=Release
-msbuild -t:RunTests -p:configuration=Release -p:DotnetPath=C:\net8-win-x86\
+msbuild -t:BuildTestsNetFx -p:TF=net462
+msbuild -t:BuildTestsNetCore
+msbuild -t:RunTests -p:DotnetPath=C:\net8-win-x86\
 ```
 
 ### Running Manual Tests
@@ -117,7 +117,7 @@ Manual Tests require the below setup to run:
   |AADSecurePrincipalSecret | (Optional) A Secret defined for a registered application which has been granted permission to the database defined in the AADPasswordConnectionString. | {Secret} |
   |AzureKeyVaultURL | (Optional) Azure Key Vault Identifier URL | `https://{keyvaultname}.vault.azure.net/` |
   |AzureKeyVaultTenantId | (Optional) The Azure Active Directory tenant (directory) Id of the service principal. | _{Tenant ID of Active Directory}_ |
-  |SupportsIntegratedSecurity | (Optional) Whether or not the USER running tests has integrated security access to the target SQL Server.| `true` OR `false`|  
+  |SupportsIntegratedSecurity | (Optional) Whether or not the USER running tests has integrated security access to the target SQL Server.| `true` OR `false`|
   |LocalDbAppName | (Optional) If Local Db Testing is supported, this property configures the name of Local DB App instance available in client environment. Empty string value disables Local Db testing. | Name of Local Db App to connect to.|
   |LocalDbSharedInstanceName | (Optional) If LocalDB testing is supported and the instance is shared, this property configures the name of the shared instance of LocalDB to connect to. | Name of shared instance of LocalDB. |
   |FileStreamDirectory | (Optional) If File Stream is enabled on SQL Server, pass local directory path to be used for setting up File Stream enabled database. |  `D:\\escaped\\absolute\\path\\to\\directory\\` |
@@ -128,15 +128,13 @@ Manual Tests require the below setup to run:
   |IsManagedInstance | (Optional) When set to `true` **TVP** related tests will use on non-Azure bs files to compare test results. this is needed when testing against Managed Instances or TVP Tests will fail on Test set 3. The default value is `false`. |
   |PowerShellPath | The full path to PowerShell.exe. This is not required if the path is present in the PATH environment variable. | `D:\\escaped\\absolute\\path\\to\\PowerShell.exe` |
 
-
 ## Example workflows using the Dotnet SDK
 
-#### Run Functional Tests
+### Run Functional Tests
 
 - Windows (`netfx x86`):
 
 ```bash
-msbuild 
 dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.Tests.csproj" -p:Platform="x86" -p:Configuration="Release" -p:TestTargetOS="Windowsnetfx" --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests"
 ```
 
@@ -147,11 +145,11 @@ dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.S
 ```
 
 - AnyCPU:
-  
+
   Project reference only builds Driver with `AnyCPU` platform, and underlying process decides to run the tests with a compatible architecture (x64, x86, ARM64).
 
   Windows (`netcoreapp`):
-  
+
 ```bash
 dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.Tests.csproj" -p:Platform="AnyCPU" -p:Configuration="Release" -p:TestTargetOS="Windowsnetcoreapp" --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests"
 ```
@@ -161,7 +159,8 @@ dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.S
 ```bash
 dotnet test "src/Microsoft.Data.SqlClient/tests/FunctionalTests/Microsoft.Data.SqlClient.Tests.csproj" -p:Platform="AnyCPU" -p:Configuration="Release" -p:TestTargetOS="Unixnetcoreapp" --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests"
 ```
-#### Run Manual Tests
+
+### Run Manual Tests
 
 - Windows (`netfx x86`):
 
@@ -199,39 +198,73 @@ dotnet test "src/Microsoft.Data.SqlClient/tests/ManualTests/Microsoft.Data.SqlCl
 dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj" -p:Platform="AnyCPU" -p:Configuration="Release" -p:TestTargetOS="Windowsnetcoreapp" --no-build -v n --filter "FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.AlwaysEncrypted.CspProviderExt.TestKeysFromCertificatesCreatedWithMultipleCryptoProviders"
 ```
 
-## Testing with Custom ReferenceType
+## Testing with Package References
+
+The MDS driver consists of several components, each of which produces its own
+NuGet package.  During development, components reference each other via
+`<ProjectReference>` properties by default.  This means that building
+and testing one component will implicitly build its project referenced
+dependencies.
 
-Tests can be built and run with custom "Reference Type" property that enables different styles of testing:
+Alternatively, the `ReferenceType` build property may be specified with a value
+of `Package`.  This will change inter-component dependencies to use
+`<PackageReference>` dependencies, and require that dependent components be
+built and packaged before building the depending component.  In this scenario,
+the root `NuGet.config` file must be updated to include the following entry
+under the `<packageSources>` element:
 
-- "Project" => Build and run tests with Microsoft.Data.SqlClient as Project Reference
-- "Package" => Build and run tests with Microsoft.Data.SqlClient as Package Reference with configured "TestMicrosoftDataSqlClientVersion" in "Versions.props" file.
+```xml
+<configuration>
+  <packageSources>
+    ...
+    <add key="local" value="packages/" />
+  </packageSources>
+</configuration>
+```
 
-> ************** IMPORTANT NOTE BEFORE PROCEEDING WITH "PACKAGE" REFERENCE TYPE ***************
-> CREATE A NUGET PACKAGE WITH BELOW COMMAND AND ADD TO LOCAL FOLDER + UPDATE NUGET CONFIG FILE TO READ FROM THAT LOCATION
->
-> ```bash
->  msbuild -p:configuration=Release
-> ```
+As a convenience, a `NuGet.config.local` file is supplied with the above
+package source already present.  You may simply copy it over `NuGet.config`
+when using `Package` references.
 
-A non-AnyCPU platform reference can only be used with package reference type. Otherwise, the specified platform will be replaced with AnyCPU in the build process.
+Then, you can specify `Package` references be used, for example:
 
-### Building Tests with Reference Type
+```bash
+cp NuGet.config.local NuGet.config
+dotnet build -t:BuildAll -p:ReferenceType=Package
+dotnet build -t:BuildAKVNetCore -p:ReferenceType=Package
+dotnet build -t:GenerateMdsPackage
+dotnet build -t:GenerateAkvPackage
+dotnet build -t:BuildTestsNetCore -p:ReferenceType=Package
+```
 
-For .NET, all 4 reference types are supported:
+The above will build the MDS and AKV components, place their NuGet packages into
+the `packages/` directory, and then build the tests using those packages.
+
+A non-AnyCPU platform reference can only be used with package reference type.
+Otherwise, the specified platform will be replaced with AnyCPU in the build
+process.
+
+### Building Tests with ReferenceType
+
+For .NET:
 
 ```bash
+# Project is the default reference type.  The below commands are equivalent:
+msbuild -t:BuildTestsNetCore
 msbuild -t:BuildTestsNetCore -p:ReferenceType=Project
-# Default setting uses Project Reference.
 
+# Package reference type:
 msbuild -t:BuildTestsNetCore -p:ReferenceType=Package
 ```
 
-For .NET Framework, below reference types are supported:
+For .NET Framework:
 
 ```bash
+# Project is the default reference type.  The below commands are equivalent:
+msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetFx -p:TF=net462 -p:ReferenceType=Project
-# Default setting uses Project Reference.
 
+# Package reference type:
 msbuild -t:BuildTestsNetFx -p:TF=net462 -p:ReferenceType=Package
 ```
 
@@ -250,26 +283,25 @@ Tests can be built and run with custom Target Frameworks. See the below examples
 ### Building Tests with custom target framework
 
 ```bash
-msbuild -t:BuildTestsNetFx -p:TF=net462
 # Build the tests for custom .NET Framework target
+msbuild -t:BuildTestsNetFx -p:TF=net462
 ```
 
 ```bash
-msbuild -t:BuildTestsNetCore -p:TF=net8.0
 # Build the tests for custom .NET target
+msbuild -t:BuildTestsNetCore -p:TF=net8.0
 ```
 
 ### Running Tests with custom target framework (traditional)
 
 ```bash
+# Run tests with custom .NET Framework target
 dotnet test -p:TargetNetFxVersion=net462 ...
-# Use above property to run Functional Tests with custom .NET Framework target
 
+# Run tests with custom .NET target
 dotnet test -p:TargetNetCoreVersion=net8.0 ...
-# Use above property to run Functional Tests with custom .NET target
 ```
 
-
 ## Using Managed SNI on Windows
 
 Managed SNI can be enabled on Windows by enabling the below AppContext switch:
@@ -294,20 +326,6 @@ When connecting to a server, if a protocol lower than TLS 1.2 is negotiated, a s
 
 `Switch.Microsoft.Data.SqlClient.SuppressInsecureTLSWarning`
 
-### Troubleshooting Docker issues
-
-There may be times where connection cannot be made to SQL Server, we found below ideas helpful:
-
-- Clear Docker images to create clean image from time-to-time, and clear docker cache if needed by running `docker system prune` in Command Prompt.
-
-- If you face `Microsoft.Data.SqlClient.SNI.dll not found` errors when debugging, try updating the below properties in the netcore\Microsoft.Data.SqlClient.csproj file and try again:
-
-  ```xml
-    <OSGroup>Unix</OSGroup>
-    <TargetsWindows>false</TargetsWindows>
-    <TargetsUnix>true</TargetsUnix>
-  ```
-
 ## Collecting Code Coverage
 
 ### Using VSTest