Run if modified: Check all commits since last build

[miracle2k]

Am I reading the code correctly that the Run if modified command is presumed to run via a pull request only, i.e. we diff against master?

I.e. as opposed to the approach of checking CIRCLE_COMPARE_URL (or the new pipeline variables) to look at the last commit a build ran for?

[roopakv]

@miracle2k yeah we currently diff against master (or any other branch you pass in as base branch). What would you diff against? I can add some other options if you think they will be useful

[miracle2k]

I found some approaches where people are diffing against whatever the last commit was a build run for (the commit CircleCI in pipelines 2.1 seems to publish as the "base revision"). I think this is basically behind the commit list that CircleCI shows you for every build.

So if I do two commits on master before pushing, it will check both of those commits.

This is basically what I was looking for, but in a nice packaged-up orb form! :)

[roopakv]

I believe swissknife does this.

let me walk you thru some scenarios

default branch of repo (base branch): master

let us say you branch from master on feature_branch_1 and add 2 commits (X1 & Y1). If you push them both up, swissknife will look at both commits for changes made)

after this you make a 3rd commit Z1 and push up, swissknife will look for changes on all X1, Y1 & Z1.

Now assume you branch feature_2 off of feature_1 and add commit D, swissknife will look at X1, Y1, Z1 and XX1 for changes.

swissknife always compares against the base branch, and never against pull requests.

 -----------A1-----> master
            |--------X1-----Y1--------Z1--> feature1
                                      |------XX1------> feature2

In the above cases feature1 whenever pushed looks at commits behind it on the same branch against A1
feature2 looks at commits ahead of master on feature2, which in this case are X1, Y1, Z1 and XX1

there is another more advanced option use-divergence-point

this is where

 -----------A1-------B1------C1---------> master
            |--------X1-----Y1-----> feature1

Here you want to only find changes in X1 & Y1 from A1 not C1, because you diverged from A1. I usually suggest that you turn this on.

Let me know what is currently missing in your case

[jin-ahn]

hi, i want to use run_if_modified multiple times consecutively to do different steps based on different patterns, however the problem is that run_if_modified completely ends the whole job if a pattern is not matched rather than continuing to the next steps. any way to change this?

[roopakv]

@jahn-mtg yeah there are a few ways I can think of doing this. The reason that this is a hard problem is that we need to explicitly communicate to the next step if the pattern matched. Presumably the best way is to use an env var.

I'm curious if trigger-workflows-if-modified might be a better fit for your use case?

it supports multiple regex checks and should solve all your problems. Let me know why modifying run_if_modified is better for you than triggering multiple workflows.

notes: https://medium.com/swissknife/monorepo-support-with-circleci-1ac5305e841e?source=friends_link&sk=643d63e4314c3732c0e73bad42fe6e55

example: https://github.com/swissknife/circle-monorepo

[jin-ahn]

Thank you for the response! But run_if_modified works better for us because its a small step as a part of a longer workflow. if we tried going the trigger-workflows-if-modified right, we could have a very long circleci.config with mostly redundant lines. Having an option flag something like "continue anyway" or something that doesn't halt the job if no match is found would better fit our needs. hopefully this i doable

[roopakv]

While I am happy to looking into adding this, i would suggest you consider splitting your yaml and using commands if the size of your circle yaml is the only reason you don't want to do this.

In the longer term it is a better fix. In the case of a continue anyway, all that the swissknife orb would do would be to set an env var and you would need a separate check in each of the steps to decide to run it or not.

Basically it will make your circle a lot more unreadable. The reason this isn't easy is because circleci doesn't allow dynamic evaluation of config yml, so any of their conditional logic cannot apply.

your steps if run if modified had continue anyway would look like this

- swissknife/run_if_modifed:
    continue_if_not_modified: true
- run:
     name: Setup DB for server test
     command: |
       if [[ "$SK_CODE_MODIFIED" == "true" ]]; then
          # write actual step here
       else
          exit 0;
       fi
- run:
     name: run server tests
     command: |
       if [[ "$SK_CODE_MODIFIED" == "true" ]]; then
          # write actual step here
       else
          exit 0;
       fi
- swissknife/run_if_modifed:
    continue_if_not_modified: true
- run:
     name: Setup for client tests
     command: |
       if [[ "$SK_CODE_MODIFIED" == "true" ]]; then
          # write actual step here
       else
          exit 0;
       fi
...

as you can see the if check now becomes repetitive and potentially something can easily be missed.

Article for splitting yml: https://blog.roopakv.com/breaking-up-circleci-yml/