-
Notifications
You must be signed in to change notification settings - Fork 136
[Dependency Cache] Enable Dependency Cache on CI #13303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6216579 to
af97b81
Compare
Collaborator
📲 You can test the changes from this Pull Request in WooCommerce-Wear Android by scanning the QR code below to install the corresponding build.
|
Collaborator
|
📲 You can test the changes from this Pull Request in WooCommerce Android by scanning the QR code below to install the corresponding build.
|
af97b81 to
6ba4e7c
Compare
6ba4e7c to
4475a6a
Compare
FYI: This change is done for testing purposes and until the below 'A8C CI Toolkit' #135 PR gets merged to 'trunk'. When that's done, the 'a8c-ci-toolkit' will be updated to '3.9.0' instead. A8C CI Toolkit PR: [Dependency Updates] [Dependency Cache] Dependency Cache on CI per Project [without GRADLE_RO_DEP_CACHE] #135 - Automattic/a8c-ci-toolkit-buildkite-plugin#135
FYI: This job will be then used by 'buildkite-ci' and configured as a 'buildkite_pipeline_schedule' with a weekly frequency. PS: The targeted 'pipeline' related jobs are: - Mobile App - Wear App - Lint - Unit Tests - Android tests
PS: The targeted 'pipeline' related jobs are: - Mobile App - Wear App - Lint - Unit Tests - Android tests PS: The 'detekt' job isn't targeted because it only takes about 20+ seconds of network request to download all the dependencies needed for this specific job, which is about the same time it take for the dependency cache to actually be restored (10+ seconds). Thus, this optimization is not really worth it for this specific job. For all those other jobs targeted, it take more than a minute of network requests to download all the dependencies needed, and as such worth the diff is worth it.
Release Notes: https://github.com/Automattic/ a8c-ci-toolkit-buildkite-plugin/releases/tag/3.9.0 Related: 4475a6a
9820220 to
3547e66
Compare
FYI: This change is done for testing purposes and until the below 'A8C CI Toolkit' #138 PR gets merged to 'trunk'. When that's done, the 'a8c-ci-toolkit' will be updated to '3.9.1' instead. A8C CI Toolkit PR: [Dependency Cache] Fix Dependency Cache #138 - Automattic/a8c-ci-toolkit-buildkite-plugin#138
1 task
Release Notes: https://github.com/Automattic/ a8c-ci-toolkit-buildkite-plugin/releases/tag/3.9.1 Related: 49d771a
wzieba
approved these changes
Jan 15, 2025
Co-authored-by: Wojciech Zięba <[email protected]>
PR Comment: https://github.com/woocommerce/woocommerce-android/pull/ 13303#discussion_r1916752265
…to ci/enable-dependency-cache-on-ci-final
When using 'single-cmd' and not using this double quotes structure it is is producing 'did not find expected key' type of errors.
…to ci/enable-dependency-cache-on-ci-final
This was referenced Jan 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
category: tooling
Anything that involves building & maintaining the project, including scripts, `Fastfile`, etc.
Core
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Project Thread: paaHJt-7CE-p2
Related: #13170 + #13122 + #13288
Depends On: a8c-ci-toolkit#135 -> a8c-ci-toolkit#138
Required By: buildkite-ci#576
Description
This PR enables dependency cache on CI for this project. 💾
The dependency cache mechanism is split between:
WooCommerce:assembleJalapenoDebug)WooCommerce-Wear:assembleJalapenoDebug)WooCommerce:lintJalapenoDebug)ssembleJalapenoDebugUnitTest assembleDebugUnitTest)assembleJalapenoDebugAndroidTest)prototype-build.sh)prototype-build.sh)lint.sh)run-unit-tests.sh)run-instrumented-tests.sh)PS.1: The
detektjob isn't targeted because it only takes about 20+ seconds of network request to download all the dependencies needed for this specific job (build + scan), which is about the same time it takes for the dependency cache to actually be restored, which is more or less about 10+ seconds (build). Thus, this optimization is not really worth it for this specific job. For all those other jobs targeted, it takes more than a minute of network requests to download all the dependencies needed (build + scan), and as such the diff is worth it.PS.2: Dependency cache isn't enabled on the release-builds.yml pipeline just yet, but just temporarily, and until we become comfortable with running this new process on the main pipeline.yml pipeline.
Future Improvements
As per this comment, me and @wzieba has been discussing on DMs about some possible future improvement, which are including but are not limited to the below list:
assemble android testtask, which depends onassembletask anyway, and would most likely resolve most of the dependencies, for all other tasks to benefit from. However, this needs further testing to make sure that running other tasks such uslintandassemble unit testare providing little to no help.init.d). Using the init script, go through all available modules for a project, and then, using that custom Gradle task, resolve all dependencies for all tasks that are available per module. We could take some inspiration from plugins such as this.dependency cachebuild to run weekly (buildkite-ci#576), make it run more often, possibly scheduling it to even run daily. However, since dependencies are not being updated too often, and, even when they do on that specific week, it would only be for a few of them. This will indeed cause some extra network requests, but most of the dependencies cached would still benefit from the existing and weekly refreshed dependency cache.Testing information
This dependency cache solution was thoroughly tested via this #13170 test-only PR, but to help you asserting that this change is indeed making an impact, plus not causing any trouble for the project, try and follow the below:
Save Cache:dependency cachejob to complete: 7m 42s#android-core-notifs):The dependency cache has been updated.(p1736943241087289-slack-C0787KRDYDC)Failure to update the dependency cache.(not tested, but should behave similarly to ☝️)Restore Cache:- 1m 45s(✅)0s1m 3s(✅)- 1m 47s(✅)0s44s(✅)- 2m 3s(✅)0s1m 35s(✅)- 2m 3s(✅)0s1m 21s(✅)+ 3m 34s(❗)0s1m 23s(✅)Note
Without: Without Dependency CacheWith: With Dependency CacheBT: Build TimeNR: Network RequestsRC: Restore CacheWin: NR (Without) - RC (With)Warning
(*): The
BT (Diff)presented above is relative, so don't read too much into it, it might not match with theWIN, which is a much more solid number. But still, it is nice to have and compare those two number together. To to be clear about that, although I tried to pick the most representative builds, without and with the dependency cache mechanism in place, there are other factors that can make any build faster, or slower for that matter (likebuild cache,firebase test lab, etc), so again, don't read too much into it.Merge Instructions
status: do not mergelabel.RELEASE-NOTES.txtif necessary. Use the "[Internal]" label for non-user-facing changes.Reviewer (or Author, in the case of optional code reviews):
Please make sure these conditions are met before approving the PR, or request changes if the PR needs improvement: