Fix Log Levels

[willarmiros]

Right now, "prod" log level is set to info: https://github.com/aws/aws-xray-daemon/blob/master/pkg/logger/log_config.go

We should configure it to "Error" level, since we state in the docs that it's the least verbose. This is arguably a breaking change since prod is the default log level, but we can discuss whether that is truly the case given the current behavior is incorrect.

We should also add a "silent" log level that suppresses all log messages.

Related: #62

[willarmiros]

Additional context: #97 changed the default log level to WARN. We decided against ERROR because that would not have emitted enough logs. However in the 3.3.0 and 3.3.1 releases, we realized this change removed all startup logging for the daemon by default which was not acceptable. So we rolled back to the original state of default logging level to info.

Going forward, we should distinguish startup logging from all other steady-state logging and ensure that startup logs are emitted regardless of log level. After startup, we should once again make the log level WARN.

[anuraaga]

@willarmiros Shouldn't we move current (noisy) INFO calls to debug and keep INFO the default? I've never seen a logging system where we have logs of a certain log level that are printed even if that log level isn't the default.

[willarmiros]

@anuraaga that's a fair callout. It's really just this bit here we'd need to change to debug to reduce pretty much all of the steady-state noise:

aws-xray-daemon/pkg/processor/batchprocessor.go

Line 126 in 814dfd8

log.Infof("Successfully sent batch of %d segments (%1.3f seconds)", len(batch), elapsed.Seconds())

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.