Skip to content

Enable notifications for TPU V0 and V1 test #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
richardsliu wants to merge 4 commits into from
Closed

Enable notifications for TPU V0 and V1 test #75

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f0eb61a
Update test-template-fastcheck.j2
richardsliu Mar 25, 2025
381eac6
add notification
richardsliu Mar 25, 2025
68fa26d
remove lines
richardsliu Mar 25, 2025
8ed80c3
test
richardsliu Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions scripts/test-template-fastcheck.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
{% set hf_home_fsx = "/fsx/hf_cache" %}

steps:

- label: ":docker: build image"
key: image-build
agents:
Expand Down Expand Up @@ -224,19 +225,36 @@ steps:
depends_on: ~

- label: "TPU V0 Test"
key: run-tpu-v0-test
depends_on: block-tpu-v0
soft_fail: true
agents:
queue: tpu_queue
commands:
- if [[ -f ".buildkite/run-tpu-test.sh" ]]; then bash .buildkite/run-tpu-test.sh; fi
commands:
- if [[ -f ".buildkite/run-tpu-test.sh" ]]; then bash .buildkite/run-tpu-test.sh; fi
- yes | docker system prune -a

- label: "TPU V0 Test Notification"
depends_on: run-tpu-v0-test
agents:
queue: tpu_queue
commands: |
if [ $$(buildkite-agent step get "outcome" --step "run-tpu-v0-test") != "passed" ]; then
cat <<- YAML | buildkite-agent pipeline upload
steps:
- label: "Notify owners about failing test"
command: echo "TPU V0 Test failed"
notify:
- email: "ricliu@google.com"
YAML
fi

- block: "Run TPU V1 Test"
key: block-tpu-v1
depends_on: ~

- label: "TPU V1 Test"
key: run-tpu-v1-test
depends_on: block-tpu-v1
soft_fail: true
agents:
Expand All @@ -245,6 +263,20 @@ steps:
- if [[ -f ".buildkite/run-tpu-v1-test.sh" ]]; then bash .buildkite/run-tpu-v1-test.sh; fi
- yes | docker system prune -a

- label: "TPU V1 Test Notification"
depends_on: run-tpu-v1-test
agents:
queue: tpu_queue
commands: |
if [ $$(buildkite-agent step get "outcome" --step "run-tpu-v1-test") != "passed" ]; then
cat <<- YAML | buildkite-agent pipeline upload
steps:
- label: "Notify owners about failing test"
command: echo "TPU V1 Test failed"
notify:
- email: "ricliu@google.com"
YAML
fi
- block: "Run GH200 Test"
depends_on: ~
key: block-gh200
Expand Down