Skip to content

[ci] Create an issue when loadtests fails#47378

Open
paulojmdias wants to merge 15 commits intoopen-telemetry:mainfrom
paulojmdias:feat/46670
Open

[ci] Create an issue when loadtests fails#47378
paulojmdias wants to merge 15 commits intoopen-telemetry:mainfrom
paulojmdias:feat/46670

Conversation

@paulojmdias
Copy link
Copy Markdown
Member

@paulojmdias paulojmdias commented Apr 4, 2026

Description

Adds a notify-failure job that creates a GitHub issue when load tests fail. Includes dedup check, performance results table, test failure output, and failed job names.

Load tests are currently failing and will be fixed in another PR.

Link to tracking issue

Relates to #46670

Testing

Tested locally against a real failed run (run ID 23989057509).

Sample issue body output

Auto-generated report for load-tests workflow.

Link to failed build: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/23989057509
Commit: 90753d1

Component(s)

testbed

Failed jobs

  • loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs)

Performance results

# Test PerformanceResults
Started: Sat, 04 Apr 2026 22:43:51 +0000

Test                                    |Result|Duration|CPU Avg%|CPU Max%|CPU Limit|RAM Avg MiB|RAM Max MiB|RAM Limit MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|--------:|----------:|----------:|------------:|---------:|-------------:|
TraceNoBackend10kSPS/NoMemoryLimit      |FAIL  |     13s|   120.6|   121.0|     0.0|         45|         96|          0|     22600|         22590|CPU consumption is 121.0%, max expected is 90%
TraceNoBackend10kSPS/MemoryLimit        |FAIL  |     23s|   113.6|   113.6|     0.0|         33|         97|          0|     18510|         18490|CPU consumption is 113.6%, max expected is 90%
Trace1kSPSWithAttrs/0*0bytes            |PASS  |     15s|    20.0|    23.0|     0.0|         66|         94|          0|     15000|         15000|
Trace1kSPSWithAttrs/100*50bytes         |PASS  |     15s|    56.2|    58.7|     0.0|         71|        100|          0|     14990|         14990|
Trace1kSPSWithAttrs/10*1000bytes        |PASS  |     15s|    40.0|    42.0|     0.0|         70|         99|          0|     14990|         14990|
Trace1kSPSWithAttrs/20*5000bytes        |PASS  |     15s|    51.9|    53.3|     0.0|         75|        105|          0|      1830|          1830|

Total duration: 97s

Test failures

--- FAIL: TestTraceNoBackend10kSPS (36.02s)
    --- FAIL: TestTraceNoBackend10kSPS/NoMemoryLimit (13.02s)
    --- FAIL: TestTraceNoBackend10kSPS/MemoryLimit (23.01s)
FAIL
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/testbed/tests	97.421s

Note: Information about any subsequent build failures that happen while this issue is open, will be added as comments with more information to this issue.

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
@paulojmdias paulojmdias added ci-cd CI, CD, testing, build issues Skip Changelog PRs that do not require a CHANGELOG.md entry labels Apr 4, 2026
paulojmdias and others added 11 commits April 4, 2026 23:13
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
…lector-contrib into feat/46670

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
…-contrib into feat/46670

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
@paulojmdias paulojmdias marked this pull request as ready for review April 5, 2026 18:17
@paulojmdias paulojmdias requested a review from a team as a code owner April 5, 2026 18:17
@paulojmdias paulojmdias requested a review from songy23 April 5, 2026 18:17
paulojmdias and others added 3 commits April 5, 2026 21:18
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
@paulojmdias
Copy link
Copy Markdown
Member Author

@songy23, I added the missing code to comment on an open issue if it is still open on subsequent failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd CI, CD, testing, build issues Skip Changelog PRs that do not require a CHANGELOG.md entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants