Skip to content

Add docs for --tag-filters #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nprizal wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add docs for --tag-filters #435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ bktec supports multiple test runners and offers various features to enhance your
| Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ |
| [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ |
| Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ |
| Filter test by tags | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Automatically retry failed test | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
| Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
Expand Down
24 changes: 23 additions & 1 deletion docs/pytest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ To integrate bktec with pytest, you need to [install and configure Buildkite Tes

```sh
export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
bktec
bktec run
```

## Configure test command
Expand Down Expand Up @@ -46,6 +46,28 @@ export BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN=tests/api
> [!TIP]
> This option accepts the pattern syntax supported by the [zzglob](https://github.com/DrJosh9000/zzglob?tab=readme-ov-file#pattern-syntax) library.

## Filter test by tags
You can filter tests to run based on `execution_tag` markers using the `BUILDKITE_TEST_ENGINE_TAG_FILTERS` environment variable or `--tag-filters` CLI option.

```py
import pytest

@pytest.mark.execution_tag('gpu_allocation', '2')
def test_my_test():
...
```

To run tests tagged with `gpu_allocation` of `2`:

```sh
export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
export BUILDKITE_TEST_ENGINE_TAG_FILTERS="gpu_allocation:2"
bktec run
```

> [!NOTE]
> `execution_tag` is a custom marker added by Buildkite Test Collector for pytest. You can add multiple tags to a test, however the tag filters only support single `key:value` pairs.

## Automatically retry failed tests
You can configure bktec to automatically retry failed tests using the `BUILDKITE_TEST_ENGINE_RETRY_COUNT` environment variable. When this variable is set to a number greater than `0`, bktec will retry each failed test up to the specified number of times, using either the default test command or the command specified in `BUILDKITE_TEST_ENGINE_TEST_CMD`.

Expand Down