Warning causes workflow to fail when `fail_on_error` is `true`

[chadfawcett]

Based on the docs, I would only expect the workflow to fail if there was an error. I'm seeing it fail when there is only a Warning. Is this the expected behaviour?

According to the reviewdog docs, a Warning should result in a "neutral" GitHub Status.

[jdkato]

Warnings should indeed result in a neutral status -- see, for example, this job with multiple warnings and a neutral status.

Can you share your .vale.ini and workflow config?

[chadfawcett]

Thanks for the response @jdkato. Here's my config setup

.vale.ini

MinAlertLevel = suggestion
StylesPath = .github/styles

[*.md]
BasedOnStyles = proselint, inclusive-language
proselint.Typography = NO
proselint.Cliches = NO
proselint.Very = NO
proselint.Cursing = NO
proselint.Currency = NO
proselint.CorporateSpeak = NO
proselint.Skunked = NO
proselint.DateSpacing = NO
proselint.DateCase = NO
proselint.Diacritical = NO
proselint.Hyperbole = NO

workflow

name: Vale Lint
on:
  pull_request:

jobs:
  prose:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Vale
        uses: errata-ai/vale-action@e9cd17e2bb87ed772b939286b7b141d7ba64ed54
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
        with:
          fail_on_error: true

Then the warning was for starting a paragraph with "but". Which is defined like so:
.github/styles/proselint/But.yml

extends: existence
message: "Do not start a paragraph with a 'but'."
level: warning
scope: paragraph
action:
  name: remove
tokens:
  - ^But

[afscrome]

I'm encountering this too - I've reproduced it at https://github.com/afscrome/vale-test/actions/runs/3349416267/jobs/5549402510 .

Whilst there are no error annotations, there are errors in the "Filtered Findings". I wonder if those are resulting in the final error status?

From my testing, the problem seems to have nothing to do with fail_on_error - even when that's false, the check itself still reporting back as "failed" - see https://github.com/afscrome/vale-test/actions/runs/3349406635 . All fail_on_error seems to do is decide if the error gets propagated up to the main job.

[jdkato]

While it's possible that I'm misconfiguring reviewdog, it seems that this isn't easy to get right:

• fail_on_error determines the return code, but not the report level (neutral, failure) which is set by level.
• We have to know the appropriate level at run time of reviewdog, but that requires knowing if Vale reported any errors.
• We can't know if Vale reported any errors at run time of reviewdog in cases where some results are filtered because that takes place after running Vale. (reviewdog Vale is slower compared to its ancestor #82)