Selecting all nodes but only tests for modified relations

[jeffkeller-einc]

I have a dbt project where creating the relations is fairly quick but running all tests is beginning to add significant time to my CI / CD pipeline. I'd like to streamline it by creating all relations but only running the tests on modified relations and their ancestors / descendants.

I know I can achieve this by sequencing dbt commands like this:

dbt seed
dbt run
dbt test --select "@state:modified" --state ...

But I would really like to capture all of this in a single selector that I can pass to dbt build so that I can:

1. Condense these steps into a single dbt build command
2. Run a single dbt retry if needed
3. Have the tests run before descendant nodes are created

I tried this:

dbt ls --exclude "state:unmodified,resource_type:test" --state ...

But this excludes all tests. I think this is because all tests are considered nodes as well and while their parent model may have changed, they have not changed themselves and so are excluded.

[apoorva-01]

You can do it in one dbt build. The reason it works with build specifically: run and test throw away everything that isn't their own resource type as the final step, but build keeps all of them. So you can union two selectors, "all models" and "tests in the modified graph":

dbt build \
  --select "resource_type:model" "resource_type:test,@state:modified" \
  --state path/to/artifacts

(add resource_type:seed resource_type:snapshot to the first group if you use them.)

Why your --exclude "state:unmodified,resource_type:test" nuked every test: comma is intersection, so that reads as "nodes that are unmodified AND a test." A data test on a modified model didn't change itself, so it's state:unmodified and it's a test, which means it matches and gets excluded. You were filtering tests by the test's own modified state, but what you actually care about is where the test sits relative to modified models.

So flip it around and select tests by graph position instead. @state:modified grabs the modified nodes plus their ancestors and descendants, and intersecting that with resource_type:test leaves just the tests hanging off that subgraph. Space unions it with resource_type:model (all models, modified or not). One build, one dbt retry if it fails, and tests still run before downstream models because build respects the DAG.

[jeffkeller-einc]

Thank you, I ended up doing something similar with selectors + dbt build.

  definition:
      union:
        - union:
            - method: fqn # all nodes
              value: "*"
            - exclude: # no tests
                - method: resource_type
                  value: test
        - union:
            - method: state # modified relations (with tests)
              value: modified