Skip to content

CI: Improve logged error visibility #2263

Open
@fallwith

Description

When the unit tests run, errors and warnings may appear in either the New Relic agent log and/or the Rails log when Rails is in scope.

Sometimes these errors are expected, because behavior that would produce a warning or an error was artificially induced under test.

Sometimes these errors are not expected, and unless a human reviews the log(s), the errors and warnings will go unnoticed.

For improved developer visibility of unexpected errors and for increased scrutiny performed by test automation, it would be ideal if errors and warnings that are found in the logs will be surfaced to the human viewer and cause a test failure when under automation.

Here's a proposal for a linear sequence of passes to take:

  1. After a test completes, quote any WARN and ERROR lines from newrelic_agent.log to STDOUT. These should be hidden unless VERBOSE_TEST_OUTPUT is enabled for now.
  2. After a test completes, also quote any WARN and ERROR lines from the relevant Rails log to STDOUT. These should be hidden unless VERBOSE_TEST_OUTPUT is enabled for now.
  3. Create an "expect this action to log with this severity" type helper that will monitor our own internal logger class's behavior AND also mark any generated log file entries to be ignored and not output to STDOUT.
  4. Once 1-3 are satisfied, consider the presence of a WARN or ERROR message to be cause for a test failure. Fail the test and output the log message(s). Do this regardless of VERBOSE_TEST_OUTPUT.

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions