HDDS-11026. Continuous Integration With GitHub Actions.#382
Open
jojochuang wants to merge 8 commits intoapache:masterfrom
Open
HDDS-11026. Continuous Integration With GitHub Actions.#382jojochuang wants to merge 8 commits intoapache:masterfrom
jojochuang wants to merge 8 commits intoapache:masterfrom
Conversation
jojochuang
requested review from
Russole,
chungen0126,
peterxcli and
sarvekshayr
adoroszlai
reviewed
Apr 11, 2026
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks a lot @jojochuang for writing this.
- Document build vs compile; drop obsolete unit job; clarify integration and native (HDDS-9242, HDDS-12734). - Replace fail-fast wording with HDDS-6464 behavior and per-job re-run. - Rephrase dependency check around LICENSE.txt; move build.sh example after table. - Update flaky/transient re-run guidance; remove deprecated workflow and outdated wiki links. Made-with: Cursor
- Add local reproduction guidance by job type (basic, dependency, integration, acceptance, kubernetes). - Point flaky debugging to flaky-test-check and repeat-acceptance-test workflows; keep IDE/tmate as optional. - Replace stale build-results mirror link; mention elek.github.io is unmaintained. - ci.yml row: include build; simplify artifact/re-run wording (no retention lecture). Made-with: Cursor
Contributor
Author
|
Thanks Attila. Updated per the suggestions. |
errose28
reviewed
Apr 14, 2026
Contributor
errose28
left a comment
errose28
left a comment
There was a problem hiding this comment.
Thanks for adding this @jojochuang and for the extensive earlier review @adoroszlai
|
|
||
| ## Run checks on your machine | ||
|
|
||
| Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. |
Contributor
There was a problem hiding this comment.
We can link within the website for this.
Suggested change
| Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. | |
| Running scripts locally catches problems before you push. You need a working dev environment first. See [Building Ozone With Maven](docs/developer-guide/build/maven). |
|
|
||
| | Rough time | Scripts | What they do | | ||
| | --- | --- | --- | | ||
| | Build step | `build.sh` | Compile Ozone | |
Contributor
There was a problem hiding this comment.
This should be a time, right?
Also If we are putting the content here there is no need to also link to it in the contributing guide.
| | Rough time | Scripts | What they do | | ||
| | --- | --- | --- | | ||
| | Build step | `build.sh` | Compile Ozone | | ||
| | Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | |
Contributor
There was a problem hiding this comment.
Suggested change
| | Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | | |
| | A few minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | |
Comment on lines
+72
to
+78
| The command below is **only an example** of running one script from the **root of your clone** (the folder that contains `hadoop-ozone/`): | ||
|
|
||
| ```bash | ||
| ./hadoop-ozone/dev-support/checks/build.sh | ||
| ``` | ||
|
|
||
| More on test styles: [Acceptance tests](./acceptance-tests) on this site. |
Contributor
There was a problem hiding this comment.
In this suggestion I had to "escape" the closing tripple quote on the bash code block to fix a github rendering error. That's not intended to be included in the final suggestion.
Suggested change
| The command below is **only an example** of running one script from the **root of your clone** (the folder that contains `hadoop-ozone/`): | |
| ```bash | |
| ./hadoop-ozone/dev-support/checks/build.sh | |
| ``` | |
| More on test styles: [Acceptance tests](./acceptance-tests) on this site. | |
| This is an example command to run the `build` check locally from the root of your fork (the directory that contains `hadoop-ozone/`): | |
| ```bash | |
| ./hadoop-ozone/dev-support/checks/build.sh | |
| \``` | |
| Other test scripts in `hadoop-ozone/dev-support/checks/` can be run similarly. To run individual acceptance tests, see [Acceptance tests](docs/developer-guide/test/acceptance-tests#running-tests-locally-using-docker-compose). |
|
|
||
| 1. **basic** — Safe to run locally without a full prior build; scripts are quick (author tags, BATS, RAT, docs, Checkstyle, PMD, SpotBugs, and similar—whatever `basic` selected for that run). | ||
| 2. **dependency / license** — Quick, but expects a **build** already (the dependency check compares against built outputs / the dependency list). | ||
| 3. **Checks that reproduce compiler or packaging issues** — Run the same **`build.sh`** (or narrower Maven command) after a normal dev build; align with the log. |
Contributor
There was a problem hiding this comment.
; align with the log
What does this mean?
| - the run is **not** from a PR, or | ||
| - the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). | ||
|
|
||
| So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. |
Contributor
There was a problem hiding this comment.
Overall on this page I think we should reduce the usage of bold font. It makes it very jerky to read and the emphasized portions aren't self explanatory enough to help with skimming. There is also no need to combine bold and monospace formatting.
Suggested change
| So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. | |
| A focused change might show fewer checks than a large refactor. That is expected. Reviewers can add `full tests needed` when the full matrix is required. If you think the wrong jobs were skipped, ask on the PR; reviewers are used to that question. |
| - **build-info** — Decides which other jobs run (selective CI). | ||
| - **build** — Performs a full build; its output is reused by later jobs. | ||
| - **compile** — Re-builds with various Java versions. It consumes the **source tarball** (release artifact) produced by **build**, not a fresh checkout of the git repository. | ||
| - **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. |
Contributor
There was a problem hiding this comment.
I would remove mention of Hugo. We should probably remove it from Ozone soon.
| - **build** — Performs a full build; its output is reused by later jobs. | ||
| - **compile** — Re-builds with various Java versions. It consumes the **source tarball** (release artifact) produced by **build**, not a fresh checkout of the git repository. | ||
| - **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. | ||
| - **dependency** — Detects whether dependencies were added or removed, as a reminder to update `LICENSE.txt` (how this is implemented—for example comparisons against `jar-report.txt`—is an internal detail). |
Contributor
There was a problem hiding this comment.
If we aren't going to specify the implementation then there's no need to mention it.
Suggested change
| - **dependency** — Detects whether dependencies were added or removed, as a reminder to update `LICENSE.txt` (how this is implemented—for example comparisons against `jar-report.txt`—is an internal detail). | |
| - **dependency** — Detects whether dependencies were added or removed, as a reminder to update `LICENSE.txt`. |
Contributor
There was a problem hiding this comment.
Agree. "how this is implemented ..." was intended as a rationale for the suggestion, not to be included in the doc.
|
|
||
| So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. | ||
|
|
||
| ## What the main CI jobs do (overview) |
Contributor
There was a problem hiding this comment.
This section should probably be moved before running checks on your machine.
| - **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). | ||
| - **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). | ||
| - **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) runs **all** JUnit tests, regardless of which submodule they live in ([HDDS-9242](https://issues.apache.org/jira/browse/HDDS-9242); often sharded in CI). Older CI had a separate **unit** job; it was removed in favor of this. | ||
| - **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. | |
| - **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges code coverage results from earlier jobs. |
sarvekshayr
reviewed
Apr 15, 2026
Contributor
There was a problem hiding this comment.
Thanks @jojochuang for adding this page. It is quite helpful.
| 2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). | ||
| 3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. | ||
|
|
||
| More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). |
Co-authored-by: Sarveksha Yeshavantha Raju <79865743+sarvekshayr@users.noreply.github.com>
- Move selective-CI section before local checks; fix GitHub Actions settings URL - Link Maven and full-tests label to site/issues; trim duplicate CONTRIBUTING pointers - Clarify local repro (FQCN for -Dtest, acceptance log example, build table times) - Soften emphasis; merge Patch Available + contributor review guidance for step 6 Made-with: Cursor
chungen0126
reviewed
Apr 29, 2026
|
|
||
| - `flaky-test-check` — defined in [`intermittent-test-check.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/intermittent-test-check.yml); runs a chosen JUnit class or method many times across parallel splits. | ||
| - `repeat-acceptance-test` — defined in [`repeat-acceptance.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/repeat-acceptance.yml); repeats acceptance tests concurrently (suite or filter). | ||
|
|
Contributor
There was a problem hiding this comment.
Thanks @jojochuang for the patch. There is an additional CI workflow called ci-with-ratis that can be triggered.
For example:
- `ci-with-ratis` - defined in [`ci-with-ratis.yml
`](https://github.com/apache/ozone/blob/master/.github/workflows/ci-with-ratis.yml); runs a full CI with a specific Ratis branch. Since Ozone is deeply integrated with Ratis, certain bugs may originate from the Ratis layer. This CI allows us to verify Ozone's stability immediately after applying fixes to Ratis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
HDDS-11026. Continuous Integration With GitHub Actions.
Please describe your PR in detail:
What is the link to the Apache Jira?
https://issues.apache.org/jira/browse/HDDS-11026
How was this patch tested?
Checked for no broken links.