Skip to content

Commit 9700b5f

Browse files
nr-ahemsathnr-ahemsath
authored
ci: Add agent configurability and convenient local testing to performance tests (#3498)
* Commit initial changes based on story description * Update compare_performance.yml * Update ChartBuilder.cs * More human readable summary for individual jobs * Don't show broken chart links * Tweaks 1. Change the agent health check to just look for "agent fully connected" so we don't have to have the agent log level set to DEBUG 2. Don't dump traffic driver and test app logs by default 3. Parameterize the python executable used to clean up the test output (part of follow-on work to make running comparisons locally convenient) * Remove plan file that got committed accidentally * Working local comparison script * Rename single test script for clarity * Rename comparison script for clarity * Update nightly comparison run to compare latest release with latest successful overnight build * Add agent configuration capability Add the ability to configure the agent (and the test app, for that matter) via env vars. Also remove the standalone "performance_tests.yml" workflow since it is redundant - its logic is now in actions/run-perf-test/action.yml, and if you want to run a single performance test case in ci, you can use compare_performance.yml with just a single run specified. * Reorganize inputs * Google AI lied to me * Semicolon-separated instead of multiline env vars * Output extra env vars if present Also stop creating unique application names for each test run that pollute our sandbox * Less app name pollution Also, remove log level setting from docker compose as it wasn't being set that way; specify it as one of the extra env var args if desired. * Rewrite individual test run script from bash to python * Fix name of script * Refactoring 1. Add a VS solution file 2. Add a pylint (python linter) config file 3. Implement linter suggestions 4. Some hand refactoring of run-perf-test.py 5. Make the agent log checks more correct and meaningful * Fix merge issue * Always quote your variables * Add README.md * Python refactoring
1 parent 425cdd1 commit 9700b5f

13 files changed

Lines changed: 1243 additions & 612 deletions

File tree

‎.github/actions/run-perf-test/action.yml‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name: Run Performance Test
22
description: >
33
Downloads the New Relic .NET agent (when requested) and runs the performance
4-
test suite via run-perf-tests.sh, then uploads the results and log artifacts.
4+
test suite via run-perf-test.py, then uploads the results and log artifacts.
55
66
inputs:
77
run-label:
@@ -37,9 +37,15 @@ inputs:
3737
collector-host:
3838
description: 'New Relic collector host (only used when attach-agent is true)'
3939
default: ''
40+
agent-env:
41+
description: 'Extra environment variables for the test app/agent, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
42+
default: ''
4043
github-token:
4144
description: 'GitHub token for downloading artifacts'
4245
required: true
46+
agent-app-name:
47+
description: 'New Relic application name to use in the test (only used when attach-agent is true)'
48+
default: 'dotnet-agent-perf-test'
4349

4450
runs:
4551
using: composite
@@ -78,7 +84,7 @@ runs:
7884
exit 1
7985
fi
8086
echo "Found workflow run ID: $WORKFLOW_RUN_ID"
81-
echo "run_id=$WORKFLOW_RUN_ID" >> $GITHUB_OUTPUT
87+
echo "run_id=$WORKFLOW_RUN_ID" >> "$GITHUB_OUTPUT"
8288
8389
- name: Download agent homefolders from release build
8490
if: inputs.attach-agent == 'true' && inputs.agent-source == 'github_release'
@@ -142,16 +148,25 @@ runs:
142148
shell: bash
143149
working-directory: ${{ env.PERF_TEST_DIR }}
144150
run: |
145-
bash run-perf-tests.sh \
151+
EXTRA_ENV_ARGS=()
152+
IFS=';' read -ra ENV_PAIRS <<< "${{ inputs.agent-env }}"
153+
for pair in "${ENV_PAIRS[@]}"; do
154+
[[ -z "$pair" ]] && continue
155+
echo "Adding extra environment variable: $pair"
156+
EXTRA_ENV_ARGS+=("--env" "$pair")
157+
done
158+
159+
python3 run-perf-test.py \
146160
--attach-agent "${{ inputs.attach-agent }}" \
147161
--agent-home "${{ github.workspace }}/${{ env.PERF_TEST_DIR }}/agent-home" \
148-
--app-name "dotnet-agent-perf-test-${{ github.run_id }}-${{ inputs.run-label }}" \
162+
--app-name "${{ inputs.agent-app-name }}" \
149163
--test-duration "${{ inputs.test-duration }}" \
150164
--locust-users "${{ inputs.locust-users }}" \
151165
--locust-spawn-rate "${{ inputs.locust-spawn-rate }}" \
152166
--dotnet-version "${{ inputs.dotnet-version }}" \
153167
--license-key "${{ inputs.license-key }}" \
154-
--collector-host "${{ inputs.collector-host }}"
168+
--collector-host "${{ inputs.collector-host }}" \
169+
"${EXTRA_ENV_ARGS[@]}"
155170
156171
# -------------------------------------------------------------------------
157172
# Upload artifacts

‎.github/workflows/compare_performance.yml‎

Lines changed: 102 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ name: Compare Performance
44
# baseline plus up to 4 agent runs) and generates a comparative report with
55
# ScottPlot charts and a markdown summary table.
66
#
7+
# Nightly scheduled run compares the latest published release against the most
8+
# recent successful scheduled all_solutions.yml build (today's nightly build).
9+
#
710
# Required secrets (when any agent run is included):
811
# PERF_TEST_ACCOUNT - New Relic license key and collector host separated by a
912
# comma. Example: XXXXXXXXXXXX,collector.newrelic.com
@@ -22,6 +25,11 @@ on:
2225
type: boolean
2326
default: true
2427
required: false
28+
no_agent_env:
29+
description: 'Extra env vars for the no-agent run, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
30+
type: string
31+
default: ''
32+
required: false
2533

2634
# --- Agent run 1 (always runs) ---
2735
run1_label:
@@ -34,6 +42,11 @@ on:
3442
type: string
3543
default: ''
3644
required: false
45+
run1_agent_env:
46+
description: 'Extra env vars for agent run 1, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
47+
type: string
48+
default: ''
49+
required: false
3750

3851
# --- Agent run 2 (optional) ---
3952
include_run2:
@@ -51,6 +64,11 @@ on:
5164
type: string
5265
default: ''
5366
required: false
67+
run2_agent_env:
68+
description: 'Extra env vars for agent run 2, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
69+
type: string
70+
default: ''
71+
required: false
5472

5573
# --- Agent run 3 (optional) ---
5674
include_run3:
@@ -68,6 +86,11 @@ on:
6886
type: string
6987
default: ''
7088
required: false
89+
run3_agent_env:
90+
description: 'Extra env vars for agent run 3, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
91+
type: string
92+
default: ''
93+
required: false
7194

7295
# --- Agent run 4 (optional) ---
7396
include_run4:
@@ -85,6 +108,11 @@ on:
85108
type: string
86109
default: ''
87110
required: false
111+
run4_agent_env:
112+
description: 'Extra env vars for agent run 4, semicolon-separated NAME=VALUE pairs (e.g. VAR1=val1;VAR2=val2)'
113+
type: string
114+
default: ''
115+
required: false
88116

89117
# --- Shared test parameters ---
90118
test_duration:
@@ -109,8 +137,9 @@ on:
109137
required: false
110138

111139
schedule:
112-
# Run nightly on weekdays: no-agent baseline vs. latest published release
113-
- cron: '0 6 * * 1-5'
140+
# Run nightly on weekdays: latest published release vs. latest scheduled nightly build
141+
# all_solutions.yml runs at 9:00 UTC and may take more than an hour, so start at 11:00 UTC
142+
- cron: '0 11 * * 1-5'
114143

115144
env:
116145
DOTNET_NOLOGO: true
@@ -123,6 +152,7 @@ concurrency:
123152

124153
permissions:
125154
contents: read
155+
actions: read
126156

127157
jobs:
128158
# ---------------------------------------------------------------------------
@@ -158,6 +188,11 @@ jobs:
158188
run4_agent_source: ${{ steps.inputs.outputs.run4_agent_source }}
159189
run4_agent_version: ${{ steps.inputs.outputs.run4_agent_version }}
160190
run4_build_run_id: ${{ steps.inputs.outputs.run4_build_run_id }}
191+
no_agent_env: ${{ steps.inputs.outputs.no_agent_env }}
192+
run1_agent_env: ${{ steps.inputs.outputs.run1_agent_env }}
193+
run2_agent_env: ${{ steps.inputs.outputs.run2_agent_env }}
194+
run3_agent_env: ${{ steps.inputs.outputs.run3_agent_env }}
195+
run4_agent_env: ${{ steps.inputs.outputs.run4_agent_env }}
161196
test_duration: ${{ steps.inputs.outputs.test_duration }}
162197
locust_users: ${{ steps.inputs.outputs.locust_users }}
163198
locust_spawn_rate: ${{ steps.inputs.outputs.locust_spawn_rate }}
@@ -169,6 +204,7 @@ jobs:
169204
id: inputs
170205
env:
171206
PERF_TEST_ACCOUNT: ${{ secrets.PERF_TEST_ACCOUNT }}
207+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
172208
run: |
173209
# Parse an agent string into agent_source / agent_version / build_run_id
174210
# and write the three outputs prefixed with $1.
@@ -177,59 +213,82 @@ jobs:
177213
local AGENT="$2"
178214
if echo "$AGENT" | grep -qE '^[0-9]+\.[0-9]'; then
179215
# Version string (e.g. "10.49.0") → github_release
180-
echo "${PREFIX}_agent_source=github_release" >> $GITHUB_OUTPUT
181-
echo "${PREFIX}_agent_version=$AGENT" >> $GITHUB_OUTPUT
182-
echo "${PREFIX}_build_run_id=" >> $GITHUB_OUTPUT
216+
echo "${PREFIX}_agent_source=github_release" >> "$GITHUB_OUTPUT"
217+
echo "${PREFIX}_agent_version=$AGENT" >> "$GITHUB_OUTPUT"
218+
echo "${PREFIX}_build_run_id=" >> "$GITHUB_OUTPUT"
183219
elif [ -n "$AGENT" ]; then
184220
# Pure digits → build_artifact run ID
185-
echo "${PREFIX}_agent_source=build_artifact" >> $GITHUB_OUTPUT
186-
echo "${PREFIX}_agent_version=" >> $GITHUB_OUTPUT
187-
echo "${PREFIX}_build_run_id=$AGENT" >> $GITHUB_OUTPUT
221+
echo "${PREFIX}_agent_source=build_artifact" >> "$GITHUB_OUTPUT"
222+
echo "${PREFIX}_agent_version=" >> "$GITHUB_OUTPUT"
223+
echo "${PREFIX}_build_run_id=$AGENT" >> "$GITHUB_OUTPUT"
188224
else
189225
# Empty → latest release
190-
echo "${PREFIX}_agent_source=github_release" >> $GITHUB_OUTPUT
191-
echo "${PREFIX}_agent_version=" >> $GITHUB_OUTPUT
192-
echo "${PREFIX}_build_run_id=" >> $GITHUB_OUTPUT
226+
echo "${PREFIX}_agent_source=github_release" >> "$GITHUB_OUTPUT"
227+
echo "${PREFIX}_agent_version=" >> "$GITHUB_OUTPUT"
228+
echo "${PREFIX}_build_run_id=" >> "$GITHUB_OUTPUT"
193229
fi
194230
}
195231
196232
if [ "${{ github.event_name }}" = "schedule" ]; then
197-
echo "include_no_agent=true" >> $GITHUB_OUTPUT
198-
echo "run1_label=latest-release" >> $GITHUB_OUTPUT
233+
# Find the most recent successful all_solutions.yml run triggered by schedule
234+
NIGHTLY_RUN_ID=$(gh run list \
235+
--workflow all_solutions.yml \
236+
--event schedule \
237+
--status success \
238+
--limit 1 \
239+
--json databaseId \
240+
--jq '.[0].databaseId')
241+
if [ -z "$NIGHTLY_RUN_ID" ]; then
242+
echo "ERROR: Could not find a recent successful scheduled all_solutions.yml run" >&2
243+
exit 1
244+
fi
245+
echo "Found nightly build run ID: $NIGHTLY_RUN_ID"
246+
echo "include_no_agent=true" >> "$GITHUB_OUTPUT"
247+
echo "run1_label=latest-release" >> "$GITHUB_OUTPUT"
199248
parse_agent "run1" ""
200-
echo "include_run2=false" >> $GITHUB_OUTPUT
201-
echo "run2_label=" >> $GITHUB_OUTPUT
202-
parse_agent "run2" ""
203-
echo "include_run3=false" >> $GITHUB_OUTPUT
204-
echo "run3_label=" >> $GITHUB_OUTPUT
249+
echo "include_run2=true" >> "$GITHUB_OUTPUT"
250+
echo "run2_label=nightly-build" >> "$GITHUB_OUTPUT"
251+
parse_agent "run2" "$NIGHTLY_RUN_ID"
252+
echo "include_run3=false" >> "$GITHUB_OUTPUT"
253+
echo "run3_label=" >> "$GITHUB_OUTPUT"
205254
parse_agent "run3" ""
206-
echo "include_run4=false" >> $GITHUB_OUTPUT
207-
echo "run4_label=" >> $GITHUB_OUTPUT
255+
echo "include_run4=false" >> "$GITHUB_OUTPUT"
256+
echo "run4_label=" >> "$GITHUB_OUTPUT"
208257
parse_agent "run4" ""
209-
echo "test_duration=2m" >> $GITHUB_OUTPUT
210-
echo "locust_users=10" >> $GITHUB_OUTPUT
211-
echo "locust_spawn_rate=2" >> $GITHUB_OUTPUT
212-
echo "dotnet_version=10.0" >> $GITHUB_OUTPUT
258+
echo "no_agent_env=" >> "$GITHUB_OUTPUT"
259+
echo "run1_agent_env=" >> "$GITHUB_OUTPUT"
260+
echo "run2_agent_env=" >> "$GITHUB_OUTPUT"
261+
echo "run3_agent_env=" >> "$GITHUB_OUTPUT"
262+
echo "run4_agent_env=" >> "$GITHUB_OUTPUT"
263+
echo "test_duration=2m" >> "$GITHUB_OUTPUT"
264+
echo "locust_users=10" >> "$GITHUB_OUTPUT"
265+
echo "locust_spawn_rate=2" >> "$GITHUB_OUTPUT"
266+
echo "dotnet_version=10.0" >> "$GITHUB_OUTPUT"
213267
else
214-
echo "include_no_agent=${{ inputs.include_no_agent }}" >> $GITHUB_OUTPUT
215-
echo "run1_label=${{ inputs.run1_label }}" >> $GITHUB_OUTPUT
268+
echo "include_no_agent=${{ inputs.include_no_agent }}" >> "$GITHUB_OUTPUT"
269+
echo "run1_label=${{ inputs.run1_label }}" >> "$GITHUB_OUTPUT"
216270
parse_agent "run1" "${{ inputs.run1_agent }}"
217-
echo "include_run2=${{ inputs.include_run2 }}" >> $GITHUB_OUTPUT
218-
echo "run2_label=${{ inputs.run2_label }}" >> $GITHUB_OUTPUT
271+
echo "include_run2=${{ inputs.include_run2 }}" >> "$GITHUB_OUTPUT"
272+
echo "run2_label=${{ inputs.run2_label }}" >> "$GITHUB_OUTPUT"
219273
parse_agent "run2" "${{ inputs.run2_agent }}"
220-
echo "include_run3=${{ inputs.include_run3 }}" >> $GITHUB_OUTPUT
221-
echo "run3_label=${{ inputs.run3_label }}" >> $GITHUB_OUTPUT
274+
echo "include_run3=${{ inputs.include_run3 }}" >> "$GITHUB_OUTPUT"
275+
echo "run3_label=${{ inputs.run3_label }}" >> "$GITHUB_OUTPUT"
222276
parse_agent "run3" "${{ inputs.run3_agent }}"
223-
echo "include_run4=${{ inputs.include_run4 }}" >> $GITHUB_OUTPUT
224-
echo "run4_label=${{ inputs.run4_label }}" >> $GITHUB_OUTPUT
277+
echo "include_run4=${{ inputs.include_run4 }}" >> "$GITHUB_OUTPUT"
278+
echo "run4_label=${{ inputs.run4_label }}" >> "$GITHUB_OUTPUT"
225279
parse_agent "run4" "${{ inputs.run4_agent }}"
226-
echo "test_duration=${{ inputs.test_duration }}" >> $GITHUB_OUTPUT
227-
echo "locust_users=${{ inputs.locust_users }}" >> $GITHUB_OUTPUT
228-
echo "locust_spawn_rate=${{ inputs.locust_spawn_rate }}" >> $GITHUB_OUTPUT
229-
echo "dotnet_version=${{ inputs.dotnet_version }}" >> $GITHUB_OUTPUT
280+
echo "no_agent_env=${{ inputs.no_agent_env }}" >> "$GITHUB_OUTPUT"
281+
echo "run1_agent_env=${{ inputs.run1_agent_env }}" >> "$GITHUB_OUTPUT"
282+
echo "run2_agent_env=${{ inputs.run2_agent_env }}" >> "$GITHUB_OUTPUT"
283+
echo "run3_agent_env=${{ inputs.run3_agent_env }}" >> "$GITHUB_OUTPUT"
284+
echo "run4_agent_env=${{ inputs.run4_agent_env }}" >> "$GITHUB_OUTPUT"
285+
echo "test_duration=${{ inputs.test_duration }}" >> "$GITHUB_OUTPUT"
286+
echo "locust_users=${{ inputs.locust_users }}" >> "$GITHUB_OUTPUT"
287+
echo "locust_spawn_rate=${{ inputs.locust_spawn_rate }}" >> "$GITHUB_OUTPUT"
288+
echo "dotnet_version=${{ inputs.dotnet_version }}" >> "$GITHUB_OUTPUT"
230289
fi
231-
echo "license_key=${PERF_TEST_ACCOUNT%%,*}" >> $GITHUB_OUTPUT
232-
echo "collector_host=${PERF_TEST_ACCOUNT##*,}" >> $GITHUB_OUTPUT
290+
echo "license_key=${PERF_TEST_ACCOUNT%%,*}" >> "$GITHUB_OUTPUT"
291+
echo "collector_host=${PERF_TEST_ACCOUNT##*,}" >> "$GITHUB_OUTPUT"
233292
234293
# ---------------------------------------------------------------------------
235294
# Parallel test runs
@@ -253,6 +312,7 @@ jobs:
253312
with:
254313
run-label: no-agent
255314
attach-agent: 'false'
315+
agent-env: ${{ needs.resolve-inputs.outputs.no_agent_env }}
256316
test-duration: ${{ needs.resolve-inputs.outputs.test_duration }}
257317
locust-users: ${{ needs.resolve-inputs.outputs.locust_users }}
258318
locust-spawn-rate: ${{ needs.resolve-inputs.outputs.locust_spawn_rate }}
@@ -282,6 +342,7 @@ jobs:
282342
agent-source: ${{ needs.resolve-inputs.outputs.run1_agent_source }}
283343
agent-version: ${{ needs.resolve-inputs.outputs.run1_agent_version }}
284344
build-run-id: ${{ needs.resolve-inputs.outputs.run1_build_run_id }}
345+
agent-env: ${{ needs.resolve-inputs.outputs.run1_agent_env }}
285346
test-duration: ${{ needs.resolve-inputs.outputs.test_duration }}
286347
locust-users: ${{ needs.resolve-inputs.outputs.locust_users }}
287348
locust-spawn-rate: ${{ needs.resolve-inputs.outputs.locust_spawn_rate }}
@@ -314,6 +375,7 @@ jobs:
314375
agent-source: ${{ needs.resolve-inputs.outputs.run2_agent_source }}
315376
agent-version: ${{ needs.resolve-inputs.outputs.run2_agent_version }}
316377
build-run-id: ${{ needs.resolve-inputs.outputs.run2_build_run_id }}
378+
agent-env: ${{ needs.resolve-inputs.outputs.run2_agent_env }}
317379
test-duration: ${{ needs.resolve-inputs.outputs.test_duration }}
318380
locust-users: ${{ needs.resolve-inputs.outputs.locust_users }}
319381
locust-spawn-rate: ${{ needs.resolve-inputs.outputs.locust_spawn_rate }}
@@ -346,6 +408,7 @@ jobs:
346408
agent-source: ${{ needs.resolve-inputs.outputs.run3_agent_source }}
347409
agent-version: ${{ needs.resolve-inputs.outputs.run3_agent_version }}
348410
build-run-id: ${{ needs.resolve-inputs.outputs.run3_build_run_id }}
411+
agent-env: ${{ needs.resolve-inputs.outputs.run3_agent_env }}
349412
test-duration: ${{ needs.resolve-inputs.outputs.test_duration }}
350413
locust-users: ${{ needs.resolve-inputs.outputs.locust_users }}
351414
locust-spawn-rate: ${{ needs.resolve-inputs.outputs.locust_spawn_rate }}
@@ -378,6 +441,7 @@ jobs:
378441
agent-source: ${{ needs.resolve-inputs.outputs.run4_agent_source }}
379442
agent-version: ${{ needs.resolve-inputs.outputs.run4_agent_version }}
380443
build-run-id: ${{ needs.resolve-inputs.outputs.run4_build_run_id }}
444+
agent-env: ${{ needs.resolve-inputs.outputs.run4_agent_env }}
381445
test-duration: ${{ needs.resolve-inputs.outputs.test_duration }}
382446
locust-users: ${{ needs.resolve-inputs.outputs.locust_users }}
383447
locust-spawn-rate: ${{ needs.resolve-inputs.outputs.locust_spawn_rate }}

0 commit comments

Comments
 (0)