Skip to content

Fix/ctf run tests quarantine #1290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Fix/ctf run tests quarantine #1290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
223a0c2
fix: ctf-run-tests quarantine
erikburt Oct 14, 2025
2f973c7
add changeset
erikburt Oct 14, 2025
f605147
test: pin ctf-run-tests version to this branch
erikburt Oct 14, 2025
55a2dc5
fix: ensure RUN_QUARANTINED_TESTS
erikburt Oct 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/empty-lamps-film.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"ctf-run-tests": minor
---

add quarantine input to gate some quarantine functionality
9 changes: 6 additions & 3 deletions .github/workflows/run-e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -835,7 +835,7 @@
- name: Run tests
id: run_tests
timeout-minutes: ${{ matrix.tests.timeout_minutes || inputs.test-timeout-minutes }}
uses: smartcontractkit/.github/actions/ctf-run-tests@ctf-run-tests/0.11.0
uses: smartcontractkit/.github/actions/ctf-run-tests@fix/ctf-run-tests-quarantine
continue-on-error: ${{ inputs.quarantine == 'true' }} # auto-quarantine will handle result
env:
DETACH_RUNNER: true
Expand All @@ -856,6 +856,7 @@
secrets.QA_AWS_REGION }}.amazonaws.com
GITHUB_API_TOKEN: ${{ steps.setup-optional-gati-token.outputs.access-token || ''}}
with:
quarantine: ${{ inputs.quarantine == 'true' }}
test_go_project_path: ${{ matrix.tests.test_go_project_path }}
test_command_to_run: ${{ matrix.tests.test_cmd }} ${{ matrix.tests.test_cmd_opts }}
test_download_vendor_packages_command: cd $(dirname ${{ matrix.tests.path }}) && go mod download
Expand Down Expand Up @@ -917,7 +918,7 @@
trunk-org-slug: chainlink
trunk-token: ${{ secrets.TRUNK_API_KEY }}
trunk-previous-step-outcome: ${{ steps.run_tests.outcome }}
trunk-job-url: ${{ format('https://github.com/{0}/actions/runs/{1}/job/{2}/attempts/{3}', github.repository, github.run_id, job.check_run_id, github.run_attempt) }}

Check failure on line 921 in .github/workflows/run-e2e-tests.yml

View workflow job for this annotation

GitHub Actions / actionlint

[actionlint] .github/workflows/run-e2e-tests.yml#L921 

property "check_run_id" is not defined in object type {container: {id: string; network: string}; services: {string => {id: string; network: string; ports: {string => string}}}; status: string} [expression]
Raw output 
.github/workflows/run-e2e-tests.yml:921:135: property "check_run_id" is not defined in object type {container: {id: string; network: string}; services: {string => {id: string; network: string; ports: {string => string}}}; status: string} [expression]
trunk-upload-only: "false"

- name: Show Otel-Collector logs
Expand Down Expand Up @@ -1142,7 +1143,7 @@
- name: Run tests
id: run_tests
timeout-minutes: ${{ matrix.tests.timeout_minutes || inputs.test-timeout-minutes }}
uses: smartcontractkit/.github/actions/ctf-run-tests@ctf-run-tests/0.11.0
uses: smartcontractkit/.github/actions/ctf-run-tests@fix/ctf-run-tests-quarantine
continue-on-error: ${{ inputs.quarantine == 'true' }} # auto-quarantine will handle result
env:
DETACH_RUNNER: true
Expand Down Expand Up @@ -1171,6 +1172,7 @@
'' }}
DATABASE_URL: postgresql://postgres:node@localhost:5432/chainlink_test?sslmode=disable
with:
quarantine: ${{ inputs.quarantine == 'true' }}
test_command_to_run: ${{ matrix.tests.test_cmd }} ${{ matrix.tests.test_cmd_opts }}
test_download_vendor_packages_command: make gomod
test_secrets_override_base64: ${{ steps.aws-test-secrets.outputs.secret_value ||
Expand Down Expand Up @@ -1215,7 +1217,7 @@
trunk-org-slug: chainlink
trunk-token: ${{ secrets.TRUNK_API_KEY }}
trunk-previous-step-outcome: ${{ steps.run_tests.outcome }}
trunk-job-url: ${{ format('https://github.com/{0}/actions/runs/{1}/job/{2}/attempts/{3}', github.repository, github.run_id, job.check_run_id, github.run_attempt) }}

Check failure on line 1220 in .github/workflows/run-e2e-tests.yml

View workflow job for this annotation

GitHub Actions / actionlint

[actionlint] .github/workflows/run-e2e-tests.yml#L1220 

property "check_run_id" is not defined in object type {container: {id: string; network: string}; services: {string => {id: string; network: string; ports: {string => string}}}; status: string} [expression]
Raw output 
.github/workflows/run-e2e-tests.yml:1220:135: property "check_run_id" is not defined in object type {container: {id: string; network: string}; services: {string => {id: string; network: string; ports: {string => string}}}; status: string} [expression]
trunk-upload-only: "false"

- name: Show Grafana URL in the test summary
Expand Down Expand Up @@ -1300,10 +1302,11 @@
- name: Run tests
id: run_tests
timeout-minutes: ${{ matrix.tests.timeout_minutes || inputs.test-timeout-minutes }}
uses: smartcontractkit/.github/actions/ctf-run-tests@ctf-run-tests/0.11.0
uses: smartcontractkit/.github/actions/ctf-run-tests@fix/ctf-run-tests-quarantine
env:
RUN_QUARANTINED_TESTS: "true" # always run quarantined tests in CI
with:
quarantine: ${{ inputs.quarantine == 'true' }}
test_go_project_path: ${{ matrix.tests.test_go_project_path }}
test_command_to_run: ${{ matrix.tests.test_cmd }} ${{ matrix.tests.test_cmd_opts }}
test_download_vendor_packages_command: cd $(dirname ${{ matrix.tests.path }}) && go mod download
Expand Down
17 changes: 9 additions & 8 deletions actions/ctf-run-tests/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,12 @@ inputs:
description: |
Whether to setup the database before running the tests. Typically used for in-memory tests.
default: "false"
quarantine:
required: false
description: |
Whether the tests are being auto-quarantined. If true, the action will not fail the workflow on test failures.
This should only be used in conjunction with "branch-out".
default: "false"

runs:
using: composite
Expand Down Expand Up @@ -287,13 +293,6 @@ runs:
shell: bash
run: ${{ inputs.build_gauntlet_command }}

# gotestloghelper gives us pretty test output
- name: Set Up gotestloghelper
shell: bash
run:
go install
github.com/smartcontractkit/chainlink-testing-framework/tools/[email protected]

- name: Set default enironment variables
shell: bash
run: |
Expand Down Expand Up @@ -349,7 +348,9 @@ runs:

- name: Run Tests
shell: bash
continue-on-error: ${{ inputs.quarantine == 'true' }}
env:
RUN_QUARANTINED_TESTS: "true" # always run quarantined tests in CI
CHAINLINK_IMAGE: ${{ inputs.cl_repo }} # TODO: to remove
CHAINLINK_VERSION: ${{ inputs.cl_image_tag }} # TODO: to remove
CHAINLINK_ENV_USER: ${{ github.actor }}
Expand All @@ -370,7 +371,7 @@ runs:
fi

# Run the test command
${{ inputs.test_command_to_run }} 2>&1 | tee /tmp/gotest.log | gotestloghelper -ci -singlepackage -hidepassingtests=false
${{ inputs.test_command_to_run }}

- name: Publish Artifacts
if: failure()
Expand Down
Loading