-
Notifications
You must be signed in to change notification settings - Fork 122
feat(logger): implement async logger #985
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
de0ff67
feat: implement async logger
JustinChengLZ 5fe074e
chore: unit tests
JustinChengLZ 4e894b9
feat: split up logs based on severity
JustinChengLZ b43b2f8
feat: add cleanup parameters in async logging
JustinChengLZ ff5da70
fix: PR comments
JustinChengLZ 6e1dffc
feat: make logger more dynamic by allowing synchronous lumberjack
JustinChengLZ 533503c
fix: PR comments
JustinChengLZ File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| /* | ||
| Copyright 2022 The Katalyst Authors. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package generic | ||
|
|
||
| type LogConfiguration struct { | ||
| CustomLogDir string | ||
| LogFileMaxSize int | ||
| LogBufferSize int | ||
| LogFileMaxAge int | ||
| LogFileMaxBackups int | ||
| } | ||
|
|
||
| func NewLogConfiguration() *LogConfiguration { | ||
| return &LogConfiguration{} | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| /* | ||
| Copyright 2022 The Katalyst Authors. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package logging | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "os" | ||
| "path" | ||
| "path/filepath" | ||
| "time" | ||
|
|
||
| "github.com/rs/zerolog/diode" | ||
| "gopkg.in/natefinch/lumberjack.v2" | ||
| "k8s.io/klog/v2" | ||
|
|
||
| "github.com/kubewharf/katalyst-core/cmd/katalyst-agent/app/agent" | ||
| "github.com/kubewharf/katalyst-core/pkg/metrics" | ||
| ) | ||
|
|
||
| type SeverityName string | ||
|
|
||
| const ( | ||
| InfoSeverity SeverityName = "INFO" | ||
| WarningSeverity SeverityName = "WARNING" | ||
| ErrorSeverity SeverityName = "ERROR" | ||
| FatalSeverity SeverityName = "FATAL" | ||
| ) | ||
|
|
||
| const ( | ||
| metricsNameNumDroppedInfoLogs = "number_of_dropped_info_logs" | ||
| metricsNameNumDroppedWarningLogs = "number_of_dropped_warning_logs" | ||
| metricsNameNumDroppedErrorLogs = "number_of_dropped_error_logs" | ||
| metricsNameNumDroppedFatalLogs = "number_of_dropped_fatal_logs" | ||
| ) | ||
|
|
||
| var ( | ||
| basePath = filepath.Base(os.Args[0]) | ||
| defaultInfoLogFileName = fmt.Sprintf("%s.%s.log", basePath, InfoSeverity) | ||
| defaultWarningLogFileName = fmt.Sprintf("%s.%s.log", basePath, WarningSeverity) | ||
| defaultErrorLogFileName = fmt.Sprintf("%s.%s.log", basePath, ErrorSeverity) | ||
| defaultFatalLogFileName = fmt.Sprintf("%s.%s.log", basePath, FatalSeverity) | ||
| ) | ||
|
|
||
| type logInfo struct { | ||
| fileName string | ||
| metricsName string | ||
| } | ||
|
|
||
| var logInfoMap = map[SeverityName]*logInfo{ | ||
| InfoSeverity: {fileName: defaultInfoLogFileName, metricsName: metricsNameNumDroppedInfoLogs}, | ||
| WarningSeverity: {fileName: defaultWarningLogFileName, metricsName: metricsNameNumDroppedWarningLogs}, | ||
| ErrorSeverity: {fileName: defaultErrorLogFileName, metricsName: metricsNameNumDroppedErrorLogs}, | ||
| FatalSeverity: {fileName: defaultFatalLogFileName, metricsName: metricsNameNumDroppedFatalLogs}, | ||
| } | ||
|
|
||
| type CustomLogger struct { | ||
| diodeWriters []diode.Writer | ||
| } | ||
|
|
||
| // NewCustomLogger creates a custom logger that can either be asynchronous or synchronous, depending on configuration. | ||
| func NewCustomLogger( | ||
| agentCtx *agent.GenericContext, logDir string, maxSizeMB, maxAge, maxBackups, bufferSize int, | ||
| ) *CustomLogger { | ||
| wrappedEmitter := agentCtx.EmitterPool.GetDefaultMetricsEmitter() | ||
|
|
||
| // If logDir is not set, we are still using klog's native logger, so we just return an empty logger without calling SetOutput() | ||
| if logDir == "" { | ||
| return &CustomLogger{} | ||
| } | ||
|
|
||
| customLogger := &CustomLogger{} | ||
| for severity, logInfo := range logInfoMap { | ||
| filePath := path.Join(logDir, logInfo.fileName) | ||
|
|
||
| // lumberjackLogger is a logger that rotates log files | ||
| lumberjackLogger := &lumberjack.Logger{ | ||
| Filename: filePath, | ||
| MaxSize: maxSizeMB, | ||
| MaxAge: maxAge, | ||
| MaxBackups: maxBackups, | ||
| } | ||
|
|
||
| // Enable async logger if buffer size is more than 0; otherwise, use synchronous lumberjack logger | ||
| if bufferSize > 0 { | ||
| // diodeWriter is a writer that stores logs in a ring buffer and asynchronously flushes them to disk | ||
| diodeWriter := diode.NewWriter(lumberjackLogger, bufferSize, 10*time.Millisecond, func(missed int) { | ||
| _ = wrappedEmitter.StoreInt64(logInfo.metricsName, int64(missed), metrics.MetricTypeNameRaw) | ||
| }) | ||
| // Overrides the default synchronous writer with the diode writer | ||
| klog.SetOutputBySeverity(string(severity), diodeWriter) | ||
| customLogger.diodeWriters = append(customLogger.diodeWriters, diodeWriter) | ||
| klog.Infof("custom async logger is enabled for the severity %s", severity) | ||
| } else { | ||
| klog.SetOutputBySeverity(string(severity), lumberjackLogger) | ||
| klog.Infof("custom sync logger is enabled for the severity %s", severity) | ||
| } | ||
| } | ||
|
|
||
| return customLogger | ||
| } | ||
|
|
||
| func (a *CustomLogger) Shutdown() { | ||
| klog.Info("[Shutdown] async writer is shutting down...") | ||
| klog.Flush() | ||
| for _, writer := range a.diodeWriters { | ||
| writer.Close() | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why using poller rather than waiter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Polling performs better with high frequency logs; for waiting, the high frequency of logs will likely cause high frequency of context switching for waking up goroutines at every log, probably leading to high overhead.