Make the output of renovate-config-validator CI / automation friendlier

[Okeanos]

Tell us more.

Renovate ships with the renovate-config-validator module that helps greatly with keeping configs valid across version changes.

I have a workflow set up that runs the following snippet after Renovate proposes PRs to ensure my Renovate config is up to date and does not use deprecated or otherwise broken functionality:

jobs:
  renovate-check:
    runs-on: ubuntu-latest
      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
        with:
          node-version-file: 'package.json'
      - name: Install Renovate
        run: npm ci
      - name: Run Renovate Config Validator
        run: |
          set +e
          validation_result=$(npx --yes --package renovate -- renovate-config-validator --strict default.json5)
          if [[ $(echo -n "${validation_result}" | grep -cF "Config validated successfully") != 1 ]]; then
            configs={$(echo "${validation_result}" | tail -n +3)}
            echo "${configs}" | jq --raw-output --sort-keys '.oldConfig' > old.json
            echo "${configs}" | jq --raw-output --sort-keys '.newConfig' > new.json
            git diff --no-index --color=always -- old.json new.json
            exit 1
          fi

As you can see in my workflow above I would like to automate some aspects of the output parsing. Right now that requires flaky output parsing by using tail or similar and then manually creating a structured document in the form of a JSON document for further processing. When config errors are present this obviously stops working altogether because my minimal script doesn't account for that (yet).

Based on the config validator documentation, some personal testing, and the current v40 release source code the following happens here as best as I understand it:

• The config validator logs everything to stdout, even the WARN and ERROR messages that appear when a migration is necessary (--strict flag) or a config error was found
• The structure being returned is in the format of:

  INFO: Validating default.json5
  WARN: Config migration necessary
  "oldConfig": {
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [ "config:base" ]
  },
  "newConfig": {
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [ "config:recommended" ]
  }
  ERROR: Found errors in configuration
  "file": "default.json5",
  "errors": [
    {
      "topic": "...",
      "message": "..."
    }
  ]
  

• There is no way to tell Renovate to give me structured output (without plain log statements) in the form of a single JSON document or similar.

I am wondering whether I am either just holding this wrong (and would greatly appreciate a pointer in a better direction) or whether changes can be made to the Renovate Config Validator to support more automation friendly output. During my research I did stumble across this discussion and the linked log remapping issue. However, I am not entirely keen on reconfiguring the log level. Also, because stdout is used all the time, I am not sure this would actually do anything for the purpose I have in mind, i.e. distinguish messages intended for humans and machine parsable output.

What I have in mind would be something like a --output=json flag that:

• either suppresses log messages or writes them to stderr or a dedicated log file
• outputs (optionally to a file) a json document like:

  {
    "file": "default.json5",
    "isValid": true,
    "migration": {
      "oldConfig": {},
      "newConfig": {}
    },
    "errors": [
      {
        "topic": "",
        "message": ""
      }
    ]
  }

• uses the exit codes as before to indicate failure & success

In my eyes this would make the renovate-config-validator more robust in a CI/CD environment and overall enhance utility.

This could, in the future be enhanced with additional output formats like github-actions to directly annotate the parsed config file using GitHub workflow commands.

[rarkins]

Please check if the env variables in https://docs.renovatebot.com/config-overview/#logging-variables apply (e.g. format)

[Okeanos]

Hey, thanks a lot for picking this thread up.

The logging configuration does definitely help. A combination of LOG_FORMAT=json and LOG_LEVEL=warn certainly means less flakiness in the output handling:

• exit code appears to work as expected
• non-relevant log messages can be suppressed
• no manual conversion to machine parsable output necessary

Thanks for pointing that out!

What'd still be awesome is, if all of the outputs (warnings, errors) would be grouped together into a single response. Even with the optimisations via logging format and level, the warnings and errors are separate documents that have to be handled manually.