Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/instructions/ado-pipelines.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,20 @@ Test timeout — `--blame-hang-timeout 10m` (configured in `build.proj` and thre
- `localFeedPath` = `$(Build.SourcesDirectory)/packages` — local NuGet feed for inter-package deps
- `packagePath` = `$(Build.SourcesDirectory)/output` — NuGet pack output

## Variable Naming — Avoid `{COMMAND}ARGUMENTS` Names

The dotnet CLI (via System.CommandLine) reads environment variables named `{COMMAND}ARGUMENTS` and silently injects their content into the parsed arguments for that subcommand. Because Azure DevOps automatically exposes all pipeline variables as uppercased environment variables, a pipeline variable named `runArguments` becomes `RUNARGUMENTS`, which `dotnet run` reads and injects into the application's `args[]` — bypassing the `--` separator.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This took many hours to figure out.


**Forbidden variable names** (any casing):
- `runArguments` — injected into `dotnet run`
- `buildArguments` — injected into `dotnet build`
- `testArguments` — injected into `dotnet test`
- Any name matching `{dotnet-subcommand}Arguments`

**Use instead**: `dotnetBuildOpts`, `dotnetRunOpts`, `stressTestArgs`, or other names that do not match the `{COMMAND}ARGUMENTS` pattern.

This affects ALL .NET SDK versions (8.0+). The injection is invisible in `[command]` log lines, making it extremely hard to diagnose. The only symptom is the application receiving unexpected arguments.

## Conventions When Editing Pipelines

- Always use templates for reusable logic — do not inline complex steps
Expand Down
68 changes: 38 additions & 30 deletions eng/pipelines/stress/stress-tests-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ parameters:
# True to enable debugging steps.
- name: debug
type: boolean
default: false
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoiding template parameter default values except at the top level.


# The prefix to prepend to the job's display name:
#
Expand All @@ -36,7 +35,6 @@ parameters:
# The verbosity level for the dotnet CLI commands.
- name: dotnetVerbosity
type: string
default: normal
values:
- quiet
- minimal
Expand All @@ -48,20 +46,18 @@ parameters:
- name: jobNameSuffix
type: string

# True to fail the job when stress tests fail. When false, test failures produce warnings
# (SucceededWithIssues) but do not fail the job.
- name: failOnTestFailure
# When true, test failures produce warnings (SucceededWithIssues) but do not fail the job.
# When false, test failures fail the job. All test steps always run regardless of this setting.
- name: warnOnTestFailure
type: boolean

# The list of .NET Framework runtimes to test against.
- name: netFrameworkTestRuntimes
type: object
default: []

# The list of .NET runtimes to test against.
- name: netTestRuntimes
type: object
default: []

# The name of the Azure Pipelines pool to use.
- name: poolName
Expand Down Expand Up @@ -105,18 +101,6 @@ jobs:
- name: project
value: $(Build.SourcesDirectory)/src/Microsoft.Data.SqlClient/tests/StressTests/SqlClient.Stress.Runner/SqlClient.Stress.Runner.csproj

# dotnet CLI arguments for build.
- name: buildArguments
value: >-
--verbosity ${{ parameters.dotnetVerbosity }}
-p:Configuration=${{ parameters.buildConfiguration }}

# dotnet CLI arguments for run.
- name: runArguments
value: >-
--verbosity ${{ parameters.dotnetVerbosity }}
--configuration ${{ parameters.buildConfiguration }}

# The contents of the config file to use for all tests. We will write this to a JSON file and
# then point to it via the STRESS_CONFIG_FILE environment variable.
- name: configContent
Expand All @@ -137,9 +121,29 @@ jobs:
}
]

# Stress test command-line arguments.
- name: testArguments
value: -a SqlClient.Stress.Tests -console
# IMPORTANT: Do NOT name pipeline variables "runArguments", "buildArguments", or
# "testArguments". ADO exposes all pipeline variables as environment variables (uppercased),
# and the dotnet CLI's System.CommandLine reads env vars matching {COMMAND}ARGUMENTS (e.g.
# RUNARGUMENTS, BUILDARGUMENTS) and silently injects their content into the parsed arguments —
# bypassing the "--" separator. This causes app arguments to contain SDK options and triggers
# unintended behavior.

# dotnet CLI options for build.
- name: dotnetBuildOpts
value: >-
--verbosity ${{ parameters.dotnetVerbosity }}
-p:Configuration=${{ parameters.buildConfiguration }}

# dotnet run options shared by all test steps (framework is appended per-step).
- name: dotnetRunOpts
value: >-
--no-build
--verbosity ${{ parameters.dotnetVerbosity }}
--configuration ${{ parameters.buildConfiguration }}

# Stress test options passed after the "--" separator.
- name: stressTestOpts
value: --assembly SqlClient.Stress.Tests --console

steps:

Expand Down Expand Up @@ -178,7 +182,7 @@ jobs:
inputs:
command: build
projects: $(project)
arguments: $(buildArguments)
arguments: ${{ variables.dotnetBuildOpts }}
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to expand static variables at pipeline expansion time, rather than runtime.


# Set a flag so test steps can distinguish a build failure from a test failure.
- pwsh: Write-Host "##vso[task.setvariable variable=buildSucceeded]true"
Expand All @@ -194,29 +198,33 @@ jobs:
# - eq(variables['buildSucceeded'], 'true') gates on the flag set above, so tests are
# skipped entirely if the build or any setup step failed (since there's nothing to run).
#
# continueOnError: ${{ not(parameters.failOnTestFailure) }}
# - When failOnTestFailure is false, continueOnError is true: a test failure marks the
# continueOnError: ${{ parameters.warnOnTestFailure }}
# - When warnOnTestFailure is true, continueOnError is true: a test failure marks the
# step and job as SucceededWithIssues (orange warning) rather than Failed.
# - When failOnTestFailure is true, continueOnError is false: a test failure fails the
# - When warnOnTestFailure is false, continueOnError is false: a test failure fails the
# job (red), though subsequent runtimes still run due to the condition above.
#
- ${{ each runtime in parameters.netTestRuntimes }}:
- task: DotNetCoreCLI@2
displayName: Test [${{ runtime }}]
condition: and(succeededOrFailed(), eq(variables['buildSucceeded'], 'true'))
continueOnError: ${{ not(parameters.failOnTestFailure) }}
continueOnError: ${{ parameters.warnOnTestFailure }}
env:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a more robust way of injecting environment variables, rather than using the -e argument.

STRESS_CONFIG_FILE: config.json
inputs:
command: run
projects: $(project)
arguments: $(runArguments) --no-build -f ${{ runtime }} -e STRESS_CONFIG_FILE=config.json -- $(testArguments)
arguments: ${{ variables.dotnetRunOpts }} -f ${{ runtime }} -- ${{ variables.stressTestOpts }}

# Run the stress tests for each .NET Framework runtime.
- ${{ each runtime in parameters.netFrameworkTestRuntimes }}:
- task: DotNetCoreCLI@2
displayName: Test [${{ runtime }}]
condition: and(succeededOrFailed(), eq(variables['buildSucceeded'], 'true'))
continueOnError: ${{ not(parameters.failOnTestFailure) }}
continueOnError: ${{ parameters.warnOnTestFailure }}
env:
STRESS_CONFIG_FILE: config.json
inputs:
command: run
projects: $(project)
arguments: $(runArguments) --no-build -f ${{ runtime }} -e STRESS_CONFIG_FILE=config.json -- $(testArguments)
arguments: ${{ variables.dotnetRunOpts }} -f ${{ runtime }} -- ${{ variables.stressTestOpts }}
12 changes: 6 additions & 6 deletions eng/pipelines/stress/stress-tests-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# ADO.Net project:
# Triggering pipeline: MDS Main CI (branch internal/main only)
# Pipeline name: sqlclient-stress
# Pipeline URL: TODO: add URL when pipeline is created
# Pipeline URL: https://dev.azure.com/SqlClientDrivers/ADO.Net/_build?definitionId=2284
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a sqlclient-stress pipeline in the ADO.Net project, so this will now trigger on successful completion of the MDS Main CI pipeline runs in that project.


# Set the pipeline run name to the day-of-year and the daily run counter.
name: $(DayOfYear)$(Rev:rr)
Expand Down Expand Up @@ -73,10 +73,10 @@ parameters:
type: boolean
default: false

# True to fail the pipeline when stress tests fail. When false (default), test failures produce
# warnings but do not fail the overall pipeline run.
- name: failOnTestFailure
displayName: Fail pipeline on test failure
# When true, test failures produce warnings (SucceededWithIssues) but do not fail the pipeline.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flips the default behaviour - now the entire pipeline run will fail if any tests steps fail.

# When false (default), test failures fail the pipeline.
- name: warnOnTestFailure
displayName: Warn (not fail) on test failure
type: boolean
default: false

Expand Down Expand Up @@ -105,5 +105,5 @@ stages:
parameters:
buildConfiguration: ${{ parameters.buildConfiguration }}
debug: ${{ parameters.debug }}
failOnTestFailure: ${{ parameters.failOnTestFailure }}
warnOnTestFailure: ${{ parameters.warnOnTestFailure }}
dotnetVerbosity: ${{ parameters.dotnetVerbosity }}
36 changes: 24 additions & 12 deletions eng/pipelines/stress/stress-tests-stage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,15 @@ parameters:
# True to enable debugging steps.
- name: debug
type: boolean
default: false

# True to fail the job when stress tests fail. When false, test failures produce warnings.
- name: failOnTestFailure
# When true, test failures produce warnings (SucceededWithIssues) but do not fail the job.
# When false, test failures fail the job. All test steps always run regardless of this setting.
- name: warnOnTestFailure
type: boolean

# The verbosity level for the dotnet CLI commands.
- name: dotnetVerbosity
type: string
default: normal
values:
- quiet
- minimal
Expand All @@ -52,11 +51,11 @@ parameters:
type: object
default: [net462]

# The list of .NET runtimes to test against. These must align with the TargetFrameworks defined
# in the stress test Directory.Build.props and the TFMs that SqlClient ships.
# The list of .NET runtimes to test against. These should include the TFMs that SqlClient ships
# as well as any upcoming runtimes being validated (e.g. net10.0 is tested but not yet shipped).
- name: netTestRuntimes
type: object
default: [net8.0, net9.0]
default: [net8.0, net9.0, net10.0]

stages:
- stage: stress_tests_stage
Expand All @@ -69,6 +68,15 @@ stages:
- name: saPassword
value: $[stageDependencies.secrets_stage.secrets_job.outputs['SaPassword.Value']]

# The 1ES pool name, determined automatically by ADO project:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a superior approach to choosing pool name. We should plan to update our other cross-project pipelines, and remove the variable group $(ci_var_poolName) approach. Our pipelines need to explicitly support all target Azure DevOps projects anyway, so no harm in hardcoding their names and pools.

Copy link
Copy Markdown
Member

@cheenamalhotra cheenamalhotra Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoding 1ES pool or image names can be tedious to manage and the more pipelines we get - the name of images can get more widespread and lost in the files.

I would suggest we use a structured variable file (single 1ES artifact) that contains only 1ES pool and image names so if you need to identify and list all the 1ES pools and images being used in this repo, you can gather them from 1 location, or if any one image needs to replaced by something new you don't need to hunt down all pipelines but can update it from one location.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# - ADO.Net -> ADO-1ES-Pool
# - public -> ADO-CI-1ES-Pool
- name: poolName
${{ if eq(variables['System.TeamProject'], 'ADO.Net') }}:
value: ADO-1ES-Pool
${{ else }}:
value: ADO-CI-1ES-Pool

jobs:

# ----------------------------------------------------------------------------------------------
Expand All @@ -80,10 +88,12 @@ stages:
debug: ${{ parameters.debug }}
displayNamePrefix: Linux
dotnetVerbosity: ${{ parameters.dotnetVerbosity }}
failOnTestFailure: ${{ parameters.failOnTestFailure }}
warnOnTestFailure: ${{ parameters.warnOnTestFailure }}
jobNameSuffix: linux
# No .NET Framework runtimes on Linux.
netFrameworkTestRuntimes: []
netTestRuntimes: ${{ parameters.netTestRuntimes }}
poolName: ADO-CI-1ES-Pool
poolName: ${{ variables.poolName }}
saPassword: $(saPassword)
sqlSetupStep:
template: /eng/pipelines/common/templates/steps/configure-sql-server-linux-step.yml@self
Expand All @@ -100,12 +110,12 @@ stages:
debug: ${{ parameters.debug }}
displayNamePrefix: Win
dotnetVerbosity: ${{ parameters.dotnetVerbosity }}
failOnTestFailure: ${{ parameters.failOnTestFailure }}
warnOnTestFailure: ${{ parameters.warnOnTestFailure }}
jobNameSuffix: windows
# Note that we include the .NET Framework runtimes for test runs on Windows.
netFrameworkTestRuntimes: ${{ parameters.netFrameworkTestRuntimes }}
netTestRuntimes: ${{ parameters.netTestRuntimes }}
poolName: ADO-CI-1ES-Pool
poolName: ${{ variables.poolName }}
saPassword: $(saPassword)
sqlSetupStep:
template: /eng/pipelines/common/templates/steps/configure-sql-server-win-step.yml@self
Expand All @@ -124,8 +134,10 @@ stages:
debug: ${{ parameters.debug }}
displayNamePrefix: macOS
dotnetVerbosity: ${{ parameters.dotnetVerbosity }}
failOnTestFailure: ${{ parameters.failOnTestFailure }}
warnOnTestFailure: ${{ parameters.warnOnTestFailure }}
jobNameSuffix: macos
# No .NET Framework runtimes on macOS.
netFrameworkTestRuntimes: []
netTestRuntimes: ${{ parameters.netTestRuntimes }}
# We don't have any 1ES Hosted Pool images for macOS, so we use a generic one from Azure
# Pipelines.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,8 @@
<CentralPackageTransitivePinningEnabled>true</CentralPackageTransitivePinningEnabled>
</PropertyGroup>

<ItemGroup>
<PackageVersion Include="System.CommandLine" Version="2.0.8" />
</ItemGroup>

</Project>
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System.Collections.Generic;

namespace Monitoring
namespace Microsoft.Data.SqlClient.Test.Stress
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave all of these classes a proper namespace.

{
public interface IMonitorLoader
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace Monitoring
namespace Microsoft.Data.SqlClient.Test.Stress
{
public class MonitorMetrics
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class GlobalExceptionHandlerAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class GlobalTestCleanupAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class GlobalTestSetupAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
public enum TestPriority
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class TestCleanupAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

using System;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Method, Inherited = true, AllowMultiple = false)]
public class TestSetupAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
using System.Collections.Generic;
using System.Text;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
[AttributeUsage(AttributeTargets.Field, Inherited = true, AllowMultiple = true)]
public class TestVariationAttribute : Attribute
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
using System.Threading;
using System.Threading.Tasks;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
public class DeadlockDetection
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
using System.Threading;
using System.Threading.Tasks;

namespace DPStressHarness
namespace Microsoft.Data.SqlClient.Test.Stress
{
public class DeadlockDetectionTaskScheduler : TaskScheduler
{
Expand Down
Loading
Loading