Tasks

[domenkozar]

We're using Nix to do congruent configuration where possible,
but in non-Nix world we often need to do convergent configuration.

We're using mostly a glue of bash at the moment, in phases like enterShell and enterTest, etc.

Problems

• there's no way to model dependencies between different snippets of code
• often we need to check whether some state is set and if not, run a script
• lack of parallelism

Requirements

• allow defining tasks in any language
• display task running task to encourage keeping tasks fast
• separate devenv abstraction, but allow running it inside devenv as a standalone tool
• allow specifying dependencies for tasks

Syntax

{ pkgs, ... }: {
  task.run = [
      "db-migrations",
  ]
  tasks.db-migrations = {
    met = "sqlx ";
    depends = [];
    meet = "bundler";
    gemfile = "Gemfile";
  };
}

Command

We'll need to find a good tool that we can use under the hood, one such tool
is Justfile see #1320

$ tasks
running db-migrations
| ...
done db-migrations ... 3ms

[bobvanderlinden]

That looks useful. Would it also be intended for devenv inner workings? For instance when requirements.txt changes, run pip install in 'enterShell' when languages.python.venv.requirements is set?

[domenkozar]

That looks useful. Would it also be intended for devenv inner workings? For instance when requirements.txt changes, run pip install in 'enterShell' when languages.python.venv.requirements is set?

Yeah exactly, I hope that a lot of the "state changed, let's do work" can be captured under this abstraction so that we can do better logging and composability.

[burke]

One possible design sketch with more explicit naming (I've never really liked 'task' for this):

{ ... }: {
  converge.constraints.add "bundler" {
    impl = "bundle-impl.rb";
    depends = [];
  };
  converge.constraints.add "db-migrations" {
    satisfied = "bin/check-db";
    satisfy = "bundle exec rake db:migrate";
    depends = [ converge.constraints.bundler ];
  };
  converge.goal = [ converge.constraints.db-migrations ];
}

This could generate some file like...

# converge.json
{
  "default": ["db-migrations"],
  "constraints": {
    "bundler": {
      "impl": "bundle-impl.rb",
    },
    "db-migrations": {
      "satisfied": "bin/check-db",
      "satisfy": "bundle exec rake db:migrate",
      "depends": ["bundler"],
    }
  }
}

And we could have a tool invoked like converge --file converge.json db-migrations

[nseetim]

I earlier mentioned a case I was stuck on where i wanted to make changes to the guest user and password and other configurations in rabbitmq that's not available as config arguments but have to be set via the command line, but I couldn't find a way of doing this within the same rabbitmq process or calling a script within devenv.sh that runs for that particular service alone, in this case rabbitmq
Does this also address such usecases?

[bobvanderlinden]

Hmm, I hadn't used Just before, but it seems it cannot handle file dependencies. See casey/just#867. This makes it less useful for cases like running pip when requirements.txt has changed.
Make might be more fitting? Or ninja?

[donk-shopify]

met = "sqlx ";
depends = [];
meet = "bundler";
gemfile = "Gemfile";

This mixes the dependency graph structure (depends) and evaluation (met, meet). That seems generally fine in terms of of devenv. I'm not sure how else that you'd determine "whether" and "what" to do at a particular node in the dependency DAG.

In the ideal case, IMO, it'd be an improvement to understand the DAG as some kind of "program". Both met and meet are two related functions that are associated with a symbol (in this case, the name of the "task), and evaluated against an attribute-set of "node data". If we were interpreting in an O-O context, then, perhaps, the symbol would be a class name to be instantiated, the attribute-set it's initial state, and met / meet are methods on that instance.

If it's not possible to represent this expression as a complete "program", then it's just make where the ability to correctly interpret the DAG depends on a multitude of other tools that we must invoke.

Perhaps the way out of this would be if the met and meet attributes referenced derivations that might be implemented within the derivations provided by devenv or from external inputs to the devenv.nix.

Something like:

{ inputs, pkgs, ... }:
{
  # rename tasks => targets because that's the traditional name
  task.targets = [
      # first thing in the array is default
      "db-migrations",
      "bundle"
  ]
  targets.db-migrations = {
    # maybe this repository uses an input that offers a better database check
    met = inputs.database.checkDatabase;
    depends = ['bundle'];
    # let's imagine that languages.ruby provided us with rake
    meet = rake.dbMigrate;
  };

  targets.bundle = {
    # again, let's imagine that languages.ruby gives us more derivations to work with
    met = ruby.bundle.state;
    meet = ruby.bundle.install
  };
}

[euphemism]

The example exactly addresses a need I have. Nice. I have a process, initial-migration, that runs every time I run devenv up and it checks state to see if migrations need to execute and no-ops otherwise. Generally, it’s only useful on the initial environment setup and is just noise in the process-compose processes list after that point.

[euphemism]

Aside: I saw a super neat looking progress indicator for tasks in some video covering the latest Zig release:

https://youtu.be/_rcD_V1oPus?t=225 / https://asciinema.org/a/661404

There's a blog post specifically covering the implementation of that here: https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html he calls it "The Zig Progress Protocol Specification".

[samrose]

It could be worth looking at https://cuelang.org/ to meet the requirements described here

Here is an example

The below is pseudocode example of creating schemas to define tasks in cue, and a generalized runner for the tasks

package main

import (
    "encoding/yaml"
    "encoding/json"
)

// Load YAML file
yamlData: yaml.Unmarshal(#readFile) & #TaggedFile

#TaggedFile: {
    #readFile: string
    if #readFile == _|_ {
        !!! "Cannot read YAML file"
    }
}

// Task schema
#Task: {
    name:  string
    cmd:   string | [...string]
    deps?: [...string]
    check?: [...string]
    env?:  [string]: string
}

// Validate tasks against schema
tasks: [string]: #Task
tasks: yamlData.tasks

// Add any additional validations or derived fields here
tasks: [string]: {
    // Example: Ensure all task names are capitalized
    name: =~"^[A-Z]"
    
}

// Command to output the task graph as JSON
command: output: {
    task: json.Marshal(tasks)
    stdout: task
}

This is an example of a yaml file cue can consume (could be json or other format too)

tasks:
  setup:
    name: "Setup Environment"
    cmd: ["./setup_script.sh"]
  build:
    name: "Build Project"
    cmd: ["make", "build"]
    deps: ["setup"]
  test:
    name: "Run Tests"
    cmd: ["make", "test"]
    deps: ["build"]
    check: ["test", "-f", "some_file"]
  deploy:
    name: "Deploy"
    cmd: ["./deploy.sh"]
    deps: ["test"]
    env:
      DEPLOY_ENV: "production"

This can be run with a Rust or Go program, output task graph, and run in parallel

(pseudo Go example but could be Rust too of course)

package main

import (
    "encoding/json"
    "fmt"
    "os/exec"
    "sync"
)

type Task struct {
    Name  string            `json:"name"`
    Cmd   []string          `json:"cmd"`
    Deps  []string          `json:"deps,omitempty"`
    Check []string          `json:"check,omitempty"`
    Env   map[string]string `json:"env,omitempty"`
}

func main() {
    // Run CUE to get task graph
    cueCmd := exec.Command("cue", "cmd", "-t", "#readFile=./tasks.yaml", "output")
    output, err := cueCmd.Output()
    if err != nil {
        fmt.Println("Error running CUE:", err)
        return
    }

    var tasks map[string]Task
    err = json.Unmarshal(output, &tasks)
    if err != nil {
        fmt.Println("Error parsing JSON:", err)
        return
    }

    // Execute tasks
    var wg sync.WaitGroup
    for name, task := range tasks {
        wg.Add(1)
        go func(name string, task Task) {
            defer wg.Done()
            executeTask(name, task, tasks)
        }(name, task)
    }
    wg.Wait()
}

func executeTask(name string, task Task, allTasks map[string]Task) {
    // Wait for dependencies
    for _, dep := range task.Deps {
        <-taskCompletionChannels[dep]
    }

    // Execute task
    fmt.Printf("Executing task: %s\n", name)
    cmd := exec.Command(task.Cmd[0], task.Cmd[1:]...)
    cmd.Env = append(cmd.Env, formatEnv(task.Env)...)
    err := cmd.Run()
    if err != nil {
        fmt.Printf("Error executing task %s: %v\n", name, err)
    }

    // Signal completion
    taskCompletionChannels[name] <- true
}

[ ... ]

[samrose]

Of course it could be possible to just cut cue out of the picture, and consume and run tasks directly in Rust too. But cue can do merging of config and some other things so I thought it worth a consideration.

[domenkozar]

I have quite a similar prototype written in Rust!

I'll try to wrap it up and it's so liberating to see we've arrived at similar results 🤯