Skip to content

[CI] Upload logs as artifacts to BuildKite#3405

Merged
kevin85421 merged 6 commits intoray-project:masterfrom
win5923:buildkite/log
Apr 29, 2025
Merged

[CI] Upload logs as artifacts to BuildKite#3405
kevin85421 merged 6 commits intoray-project:masterfrom
win5923:buildkite/log

Conversation

@win5923
Copy link
Member

@win5923 win5923 commented Apr 16, 2025 •
edited
Loading

Why are these changes needed?

#3109

I made the test fail to see if any artifacts would be generated — looks good!

Result:
image

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8147#01963f30-29e8-4225-be66-66a6efa0f516/1102-1161

Related issue number

Closes #3109

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@win5923 win5923 marked this pull request as draft April 16, 2025 15:01
@win5923 win5923 changed the title [CI] Upload logs as artifacts to BuildKite [Do not merge][CI] Upload logs as artifacts to BuildKite Apr 16, 2025
@win5923 win5923 force-pushed the buildkite/log branch 2 times, most recently from b8537b4 to e225632 Compare April 16, 2025 15:39
@win5923 win5923 changed the title [Do not merge][CI] Upload logs as artifacts to BuildKite [CI] Upload logs as artifacts to BuildKite Apr 16, 2025
@win5923 win5923 marked this pull request as ready for review April 16, 2025 15:42
@win5923
Copy link
Member Author

win5923 commented Apr 16, 2025 •
edited
Loading

@MortalHappiness PTAL, thanks!

Also, I noticed that Test E2E Operator Version Upgrade (v1.3.0) requires Go v1.23 to run. Should I open a separate PR for that, or handle it here?

Edited: I create another PR to address this #3406

image

@MortalHappiness
Copy link
Member

MortalHappiness commented Apr 17, 2025

We should also upload ${{ env.KUBERAY_TEST_OUTPUT_DIR }}/**/*.log as mentioned in the issue. Please zip them and then upload it as artifact. Thanks.

@win5923 win5923 marked this pull request as draft April 17, 2025 14:07
@win5923 win5923 force-pushed the buildkite/log branch 20 times, most recently from 36e357e to 70e6e2b Compare April 17, 2025 18:06
@win5923
Fix pattern contains path separator
56d9373 
Signed-off-by: win5923 <ken89@kimo.com>
@win5923 win5923 marked this pull request as ready for review April 26, 2025 15:26
@win5923
Copy link
Member Author

win5923 commented Apr 26, 2025 •
edited
Loading

Hi @MortalHappiness, I've made the necessary fixes, I didn't expect that setting variables in Buildkite requires using $$.

result:

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8548#0196729a-1020-4297-8809-490e7688731f

image

@MortalHappiness
Copy link
Member

MortalHappiness commented Apr 26, 2025

I found a nested folder workdir/ray-operator/tmp in the tar file. These nested folders are not neccessary. KUBERAY_TEST_OUTPUT_DIR should be the top-level folder.

@win5923 win5923 force-pushed the buildkite/log branch 2 times, most recently from 289afb7 to da19dc9 Compare April 26, 2025 17:30
@win5923
Use relative path
56631f0 
Signed-off-by: win5923 <ken89@kimo.com>
@win5923
Copy link
Member Author

win5923 commented Apr 26, 2025

I found a nested folder workdir/ray-operator/tmp in the tar file. These nested folders are not neccessary. KUBERAY_TEST_OUTPUT_DIR should be the top-level folder.

Fixed in 56631f0

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8554#01967328-05a0-439a-af31-fda817b01d9a

MortalHappiness
MortalHappiness reviewed Apr 27, 2025
View reviewed changes
@win5923 win5923 marked this pull request as draft April 27, 2025 11:16
@win5923
remove the variable
f8694e9 
Signed-off-by: win5923 <ken89@kimo.com>
@win5923 win5923 marked this pull request as ready for review April 27, 2025 11:35
MortalHappiness
MortalHappiness reviewed Apr 29, 2025
View reviewed changes
.buildkite/test-e2e.yml Outdated
- echo "--- END:e2e rayservice (nightly operator) tests finished"
- mkdir -p "$(pwd)/tmp" && export KUBERAY_TEST_OUTPUT_DIR=$(pwd)/tmp
- echo "KUBERAY_TEST_OUTPUT_DIR=$$KUBERAY_TEST_OUTPUT_DIR"
- KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v ./test/e2e 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && find "$$KUBERAY_TEST_OUTPUT_DIR" -name "*.log" | tar --transform='s|^.*/tmp/||' -cf /artifact-mount/e2e-log.tar -T - && exit 1)
Copy link
Member

@MortalHappiness MortalHappiness Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use --transform is not correct because KUBERAY_TEST_OUTPUT_DIR may not always be tmp. I think you need to find a flag to use that folder as archive root. You can also try zip command.

Copy link
Member Author

@win5923 win5923 Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can i just cd into KUBERAY_TEST_OUTPUT_DIR and zip the *.log files?

Copy link
Member Author

@win5923 win5923 Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result: https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8618#01967f55-e074-43be-918a-f7fc043da086

@win5923 win5923 force-pushed the buildkite/log branch 3 times, most recently from d99d698 to 909876f Compare April 29, 2025 02:38
remove transform command
dfd5ad6 
Signed-off-by: blocka-wan <blocka_wan@trendmicro.com>
@kevin85421 kevin85421 merged commit 0561ba1 into ray-project:master Apr 29, 2025
23 checks passed
@win5923 win5923 deleted the buildkite/log branch April 29, 2025 08:43
laurafitzgerald pushed a commit to laurafitzgerald/kuberay that referenced this pull request May 7, 2025
@win5923 @laurafitzgerald
[CI] Upload logs as artifacts to BuildKite (ray-project#3405)
25d18f9
laurafitzgerald pushed a commit to laurafitzgerald/kuberay that referenced this pull request May 8, 2025
@win5923 @laurafitzgerald
[CI] Upload logs as artifacts to BuildKite (ray-project#3405)
d73704c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevin85421 kevin85421 kevin85421 approved these changes

@MortalHappiness MortalHappiness MortalHappiness approved these changes

Assignees

@kevin85421 kevin85421

@MortalHappiness MortalHappiness

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[CI] Upload logs as artifacts to BuildKite

3 participants

@win5923 @MortalHappiness @kevin85421