Skip to content

Fix JDK matrix pipeline after configurable it split #17461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 8, 2025
Merged

Fix JDK matrix pipeline after configurable it split #17461

merged 2 commits into from
Apr 8, 2025

Conversation

dliappis
Copy link
Contributor

@dliappis dliappis commented Apr 1, 2025 •
edited
Loading

Release notes

[rn:skip]

What does this PR do?

PR #17219 introduced configurable split quantities for IT tests, which resulted in broken JDK matrix pipelines (e.g. as seen via the elastic internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.

)

This PR fixes the above problem.

Why is it important/What is the impact to the user?

Restores the functionality of the JDK (Windows/Linux) matrix pipelines.

How to test this PR locally

Linux

BUILDKITE_PIPELINE_NAME="Logstash Linux JDK matrix pipeline" MATRIX_OSES="ubuntu-22.04" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

produces:

Output from generate-steps for Linux 
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: ubuntu-22.04/adoptiumjdk_21
    key: ubuntu-22_04_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 "### Group: ubuntu-22.04 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-java-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport ENABLE_SONARQUBE=\"false\"\nci/unit_tests.sh java\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Java Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Java Unit Test |\n\"\nfi\n      "
      - label: Ruby Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-ruby-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/unit_tests.sh ruby\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Ruby Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Ruby Unit Test |\n\"\nfi\n      "
      - label: Integration Tests - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 1/3 |\n\"\nfi\n      "
      - label: Integration Tests - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 2/3 |\n\"\nfi\n      "
      - label: Integration Tests - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 3/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 1/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 2/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 3/3 |\n\"\nfi\n      "
      - label: x-pack unit tests
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/unit_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack unit tests |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack unit tests |\n\"\nfi\n      "
      - label: x-pack integration
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-integration
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/integration_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack integration |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack integration |\n\"\nfi\n      "

Windows

BUILDKITE_PIPELINE_NAME="Logstash Windows JDK matrix pipeline" MATRIX_OSES="windows-2025" MATRIX_JDKS="adoptiumjdk_21" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

results in the following output:

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: windows-2025/adoptiumjdk_21
    key: windows-2025_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: windows-2025-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=windows-2025_adoptiumjdk_21 "### Group: windows-2025 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: windows-2025_adoptiumjdk_21-java-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Java Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 java\" -Annotate\n        "
      - label: Ruby Unit Test
        key: windows-2025_adoptiumjdk_21-ruby-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Ruby Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 ruby\" -Annotate\n        "

Related issues

PR #17219

@dliappis
Fix JDK matrix pipeline after configurable it split
9bdcd7a 
PR elastic#17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem.
@mergify Mergify
Copy link

mergify bot commented Apr 1, 2025

This pull request does not have a backport label. Could you fix it @dliappis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
  • backport-8.x is the label to automatically backport to the 8.x branch.
  • If no backport is necessary, please add the backport-skip label

dliappis
dliappis commented Apr 1, 2025
View reviewed changes
.buildkite/scripts/jdk-matrix-tests/generate-steps.py
def pq_integration_test_parts(self, parts) -> list[JobRetValues]:
return list(map(lambda idx: pq_integration_tests(self, idx+1, parts), range(parts))
def pq_integration_test_parts(self, parts) -> list[partial[JobRetValues]]:
return [partial(self.pq_integration_tests, part=idx+1, parts=parts) for idx in range(parts)]
Copy link
Contributor Author

@dliappis dliappis Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(links below are Elastic internal)

Linux tests using this PR: https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/449
Windows tests using this PR: https://buildkite.com/elastic/logstash-windows-jdk-matrix-pipeline/builds/398

@dliappis dliappis requested a review from yaauie April 1, 2025 09:28
@dliappis dliappis self-assigned this Apr 1, 2025
@dliappis dliappis added ci backport-8.17 Automated backport with mergify backport-9.0 Automated backport to the 9.0 branch with mergify backport-8.16 Automated backport with mergify backport-8.x Automated backport to the 8.x branch with mergify backport-8.18 Automated backport with mergify labels Apr 1, 2025
@dliappis dliappis marked this pull request as ready for review April 1, 2025 09:29
@dliappis
Copy link
Contributor Author

dliappis commented Apr 1, 2025

For reviewers: please advise regarding backport to 9.0, is this ok?

@jsvd
Copy link
Member

jsvd commented Apr 1, 2025

Sorry we missed this @dliappis , @andsel anything we want to take from this PR to add to the merged #17462?

@dliappis
Copy link
Contributor Author

dliappis commented Apr 1, 2025

Sorry we missed this @dliappis , @andsel anything we want to take from this PR to add to the merged #17462?

Thanks @jsvd / @andsel for the quick comment!

This purely boils down to preference/readability/more idiomatic Python; both

as things are now after #17462

        return list(map(lambda idx: integration_tests(self, idx+1, parts), range(parts))

or my suggestion

        return [partial(self.integration_tests, part=idx+1, parts=parts) for idx in range(parts)]

are identical in functionality.

Personally I find list comprehensions more concise and readable (as mentioned in the above official Python docs) as well as partial functions also a very readable way to create a new function from an existing one with some parts of the arguments preset, but this is matter of personal preference.

Therefore feel free to close this PR if you prefer.

@andsel
Copy link
Contributor

andsel commented Apr 1, 2025

@jsvd I'm not able to express a preference here, it's matter of returning list of JobRetValues to list of partial functions that return JobRetValues. We could resolve conflicts and accept this. I trust @dliappis Python taste

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

History

cc @dliappis

@dliappis dliappis requested a review from andsel April 7, 2025 13:03
andsel
andsel approved these changes Apr 7, 2025
View reviewed changes
Copy link
Contributor

@andsel andsel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dliappis dliappis merged commit b9469e0 into elastic:main Apr 8, 2025
9 checks passed
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
e909fe1 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
fb7e0ec 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
9d5e282 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
ade0d37 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
181c52a 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
@dliappis
Copy link
Contributor Author

dliappis commented Apr 8, 2025 •
edited
Loading

Merged and successfully tested (elastic internal links: linux / windows)

dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17514)
fcf313a 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17510)
d136e50 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17513)
e0d5464 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17512)
cde5786 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17511)
5f475f0 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <[email protected]>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yaauie yaauie yaauie approved these changes

@andsel andsel andsel approved these changes

Assignees

@dliappis dliappis

Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify backport-8.18 Automated backport with mergify backport-9.0 Automated backport to the 9.0 branch with mergify ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dliappis @jsvd @andsel @elasticmachine @yaauie