Find changed tasks prior to execution

[jakeleventhal]

What is the improvement or update you wish to see?

turbo-ignore should allow me to specify all packages for any command and allow me to see if I need to run CI.

Is there any context that might help us understand?

My current CI pipeline looks like this:

1. Clone repo
2. Install dependencies
3. Link to Vercel
4. turbo lint test tsc

I want to be able to add a step immediately after step 1 where I can simply run turbo-ignore lint test tsc. Idea is that I would be able to just see "does this already exist in my remote cache". I want it set up to check the remote cache rather than diff from main because if I run CI locally, I don't want to redo it in CI.

I tried to kind of hack this together myself using a script like this:

#!/bin/bash

turboVersion=$(jq -r .devDependencies.turbo < "package.json")

MISS_COUNT=$(npx -y turbo@$turboVersion lint test tsc --dry=json | jq '[.tasks[] | select(.cache.status != "HIT")] | length')
echo "Number of cache misses: $MISS_COUNT"

But then I was running into the issues in this issue. I would run turbo lint test tsc and get FULL TURBO but then this would show dozens of misses.

TL;DR, I want a single CLI command that just tells me yes or no if all my outputs are cached.

Does the docs page already exist? Please link to it.

The docs are limited here.

[dBianchii]

--dry will output a list of the task per package, even if the package doesn't have this task listed.
In order to just filter for real existent tasks, you need to filter by command != <NONEXISTENT>:

`turbo lint test tsc --dry=json | jq '[.tasks[] | select(.command != "")]'

I think the --dry docs need to be updated to better explain these things.

[anthonyshew]

For visibility: @dBianchii sent this reproduction: https://github.com/dBianchii/turborepo-dry-bugs

[anthonyshew]

@jakeleventhal @dBianchii Would something like this with turbo query work?

query GetInvalidations($sha: String!, $filter: PackagePredicate) {
  affectedPackages(base: $sha, filter: $filter) {
    length
    items {
      name
      path
    }
  }
}

{
  "sha": "HEAD~2",
  "filter": {
    "or": [
      {
        "has": {
          "field": "TASK_NAME",
          "value": "unit-test"
        }
      },
      {
        "has": {
          "field": "TASK_NAME",
          "value": "build"
        }
      }
    ]
  }
}

[jakeleventhal]

@anthonyshew This could work, but personally this seems pretty overkill to be using GraphQL for something this lightweight. Seems like a CLI command much more closely matching the real query would be better (turbo-check lint test tsc or something)

[anthonyshew]

I'm trying to sort out if this raw query is what would meet your needs. It's still not clear what the expected outcomes are for given scenarios. Once there's a generalizable and useful user story, we can figure out if makes sense to have dedicated API and what it would be.

[dBianchii]

@anthonyshew Thanks for the feedback. turbo query seems to check for affected runs given a specific reference SHA. Seems to be similar to how —dry works. I played a bit with it in GraphiQL IDE, but I couldn't get the behavior I wanted.

For our needs, we cannot use SHAs. We want a simple way to check if a lint or deploy run exists in our cache.

We can't know the SHA of our last successful deployment.

Currently, when we merge a PR, a GH action runs for every app, executing "pnpm turbo deploy:production". This deploys our app to vercel.

It would be great if before we even run this, we could ask "does this deploy:production run exist in our cache?" (either by --dry or by turbo query or however) without specifying a SHA at all.

Is there a way to do this via --dry or via turbo query?

[anthonyshew]

An update: We've talked this through as a team several times now and we're realizing we need a new primitive built on top of turbo query.

turbo-ignore doesn't really do what folks expect it to do, and the nuances make it hard to use and hard to explain. It was meant for a more constrained use case than what most want to use it for (as seen here), and the DX isn't as good as it should be. Normally, we would iterate here, but those aforementioned difficulties mean that existing usages of turbo-ignore are very likely to break in subtle and difficult to understand ways, so starting from scratch with a purpose-built alternative is preferable.

We do have this improvement on our roadmap, and we're looking forward to getting it shipped.

[jakeleventhal]

Thanks for the update @anthonyshew
Excited for this feature

[anthonyshew]

Because this is now evolving into a feature request, I'm going to bounce it out to Discussions per our way of work (issues in Issues, feature requests in Discussions).