notify maintainers on evaluation errors

[milahu]

it would be nice to have a push mechanism,
to notify maintainers on evaluation errors

since the evaluation is done many times per day,
only the first error should be reported

the readme says

You can find out if your evaluation succeeded by checking the latest build job.

for example, the build job says

INFO:nur.update:Evaluate repository milahu
error: while evaluating 'callPackageWith' at /nix/store/8qkck0vfjjgc4fjxkgvj3vc68dq3pj9j-nixexprs.tar.xz/lib/customisation.nix:117:35, called from /nix/store/7ffdwrywanjwqlg7l4ndvv7j3ygjyqbn-source/default.nix:19:24:
while evaluating 'makeOverridable' at /nix/store/8qkck0vfjjgc4fjxkgvj3vc68dq3pj9j-nixexprs.tar.xz/lib/customisation.nix:67:24, called from /nix/store/8qkck0vfjjgc4fjxkgvj3vc68dq3pj9j-nixexprs.tar.xz/lib/customisation.nix:121:8:
getting status of '/nix/store/7ffdwrywanjwqlg7l4ndvv7j3ygjyqbn-source/pkgs/openhantek': No such file or directory

expected result:
a bot should create an issue on my tracker
https://github.com/milahu/nur-packages/issues

tags: notification, debug

[Mic92]

Yes that would be nice. Is there a github action for that?

[milahu]

Is there a github action for that?

i guess this belongs in nur/update.py

        logger.info(f"Evaluate repository {repo.name}")
        proc = subprocess.Popen(
            cmd, env=dict(PATH=os.environ["PATH"]), stdout=subprocess.DEVNULL
        )
        try:
            res = proc.wait(10)
        except subprocess.TimeoutExpired:
            raise EvalError(f"evaluation for {repo.name} timed out of after 10 seconds")
        if res != 0:
            raise EvalError(f"{repo.name} does not evaluate:\n$ {' '.join(cmd)}")

so we need:

• capture the output of the subprocess, correctly interleave stdout and stderr
• a github api client in python, for example PyGithub to create issue
• a github account for the bot -> @nur-bot?
• hide the access token -> github repo config?
• avoid duplicate issues -> get issue?
• handle repos without issue trackers -> create issue in NUR/issues and mention the maintainer
• allow maintainers to turn off the notifications -> repos.json? default on?

concept:

#!/usr/bin/env python3

# https://github.com/nix-community/NUR/blob/master/nur/update.py

import subprocess
import os
from github import Github # https://github.com/PyGithub/PyGithub

bot_name = "nur-bot" # https://github.com/nur-bot
bot_token = os.environ["GITHUB_BOT_TOKEN"] # access token
# TODO set token in github config

eval_timeout = 10 # seconds

github = Github(bot_token)

# TODO loop values from repos.json
# sample: https://github.com/milahu/nur-packages
repo_owner = "milahu"
repo_name = "nur-packages"
repo_version = "TODO"

cmd = [
    "bash",
    "test.sh"
]

proc = subprocess.Popen(
    cmd, env=dict(PATH=os.environ["PATH"]),
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)

timeout = False
nonzero = False

try:
    proc.wait(eval_timeout)
except subprocess.TimeoutExpired:
    timeout = True

if proc.returncode != 0:
    nonzero = True

if timeout or nonzero:
    output = proc.stdout.read().decode() # stdout and stderr are interleaved

    # send output to logfile
    print(output)

    # notify maintainer
    # TODO check if maintainer wants to receive notifications -> repos.json
    # check if we already sent a notification for this error
    repo = github.get_repo(f"{repo_owner}/{repo_name}")
    last_issue = repo.get_issues(creator=bot_name, sort="newest")[0]
    issue_title = f"evaluation failed for version {repo_version}"
    if issue_title != last_issue.title:
        # we have not yet notified the maintainer of this error
        # create a new issue on the maintainer's repo
        issue_body = f"evaluation failed.\n\ntimeout: {"yes" if timeout else "no"}\nreturn code: {proc.returncode}\n\noutput:\n\n```\n" + output.strip() + "\n```\n"
        repo.create_issue(title=issue_title, body=issue_body)
        # TODO handle "no issue tracker"

if timeout:
    raise EvalError(f"evaluation for {{repo.name}} timed out of after {eval_timeout} seconds")

if nonzero:
    raise EvalError(f"{{repo.name}} does not evaluate:\n$ {' '.join(cmd)}")

[Mic92]

Btw. you can also check evaluation in your own repo using: https://github.com/nix-community/nur-packages-template/blob/cc8c7823f2280d6216c521efb3814312eeae483e/.github/workflows/build.yml#L57

[milahu]

yepp:

nix-env -f . -qa \* --meta --xml \
  --allowed-uris https://static.rust-lang.org \
  --option restrict-eval true \
  --option allow-import-from-derivation true \
  --drv-path --show-trace \
  -I nixpkgs=$(nix-instantiate --find-file nixpkgs) \
  -I ./ \
  >/dev/null

edit: error: access to path '/nix/store/j777ivm3g8b3nq6vii1bccfbahsa9cl3-nixos-21.11pre309670.253aecf69ed/nixos' is forbidden in restricted mode is solved by replacing -I $PWD with -I ./

[Rhys-T]

Just a heads up that this 'Check evaluation' nix-env command doesn't seem to catch everything. In particular, I recently had a mistake in my repo where one package's meta.homepage was failing to evaluate, because it was being inherited from the wrong attrset. The nix-env step in my repo's workflow just silently left the homepage out of its XML output, but it caused an actual eval error on NUR's side, causing my repo to disappear from the site. Still helps with the more common cases, though.