Add a --checkout option to tweak which jobs are run #8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Many of our .buildbot_dockerfile_* files use `docker buildx` as part of the build process. If the host machine does not have this installed then tryci jobs fail much later on with very strange error messages. We should catch this early and make it clear what's going on.
|
Does this mean we should close #7 ? |
|
That one is still useful (I'm using a hacky version of it on b14). If/when you're happy with this I'll rebase it against this and update it to address your comment. |
This is a big change, but one I have been wanting for a while. It fixes a huge limitation of TryCI which is that it could only fire off CI jobs from the working tree of a repository. A lot of the time this was good enough, but it can lead to subtle bugs where TryCI behaves differently to our CI server. It's easy to accidentally have dirty files sent over as part of the build context and influence the CI result (even if they were .gitignored). Now we can use TryCI to run CI jobs at specific git refs. This works with locally cloned repos and with remote URLs. Here are some examples: ``` tryci --checkout "HEAD" tryci --checkout "feature-branch" tryci --checkout "origin/master" tryci --checkout "f5a7b2" tryci --checkout "https://github.com/ykjit/yk#trying" ``` When the '--checkout <ref>' option is used, TryCI will checkout <ref> in a tempdir which becomes the build context for that CI job. It also sets this temp repo up to emulate buildbot on bencher13 as closely as possible (cloning with a depth of 100; setting the correct remote-urls etc). To ensure that docker build caching still works, this commit also adjusts the naming scheme for docker image tags. Checkouts of local builds will have the following format: <user>-local-<rootdir>:<ref>-<dockerfile_suffix> So running 'tryci --checkout "f5a7b2"' in a local `yk` repo will build the following image: jake-local-yk:f5a7b2-debian Remote builds will have the following format: <user>-github.com_<org>_<repo>:<ref>-<dockerfile_suffix> So 'tryci --checkout "https://github.com/ykjit/yk#trying"' would be: jake-github.com_ykjit_yk:trying-debian This way TryCI checkouts which are not based on the working tree will not invalidate the caches of previous jobs. Perhaps the biggest win from this is that you can fire off a fresh CI job on a machine which you don't even have the repo cloned on. You can also do this in parallel because no two builds copy context from the same directory (i.e. your working tree). This makes bisecting so much easier.
jacob-hughes
force-pushed
the
ci_checkouts
branch
from
b6616c7 to
a1dad5d
Compare
|
Force-pushed a minor fix where I forgot to update the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a big change, but one I have been wanting for a while. It fixes a huge limitation of TryCI which is that it could only fire off CI jobs from the working tree of a repository.
A lot of the time this was good enough, but it can lead to subtle bugs where TryCI behaves differently to our CI server. It's easy to accidentally have dirty files sent over as part of the build context and influence the CI result (even if they were .gitignored).
Now we can use TryCI to run CI jobs at specific git refs. This works with locally cloned repos and with remote URLs. Here are some examples:
When the '--checkout ' option is used, TryCI will checkout in a tempdir which becomes the build context for that CI job. It also sets this temp repo up to emulate buildbot on bencher13 as closely as possible (cloning with a depth of 100; setting the correct remote-urls etc).
To ensure that docker build caching still works, this commit also adjusts the naming scheme for docker image tags. Checkouts of local builds will have the following format:
<user>-local-<rootdir>:<ref>-<dockerfile_suffix>So running 'tryci --checkout "f5a7b2"' in a local
ykrepo will build the following image:jake-local-yk:f5a7b2-debianRemote builds will have the following format:
<user>-github.com_<org>_<repo>:<ref>-<dockerfile_suffix>So 'tryci --checkout "https://github.com/ykjit/yk#trying"' would be:
jake-github.com_ykjit_yk:trying-debianThis way TryCI checkouts which are not based on the working tree will not invalidate the caches of previous jobs.
Perhaps the biggest win from this is that you can fire off a fresh CI job on a machine which you don't even have the repo cloned on. You can also do this in parallel because no two builds copy context from the same directory (i.e. your working tree). This makes bisecting so much easier.