-
Notifications
You must be signed in to change notification settings - Fork 285
refactor: remove|optimize logging #23231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
PR Compliance Guide 🔍Below is a summary of compliance checks for this PR:
Compliance status legend🟢 - Fully Compliant🟡 - Partial Compliant 🔴 - Not Compliant ⚪ - Requires Further Human Verification 🏷️ - Compliance label |
|||||||||||||||||||||||
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||||||
Merge Queue Status✅ The pull request has been merged at afb6780 This pull request spent 56 minutes 16 seconds in the queue, including 56 minutes 7 seconds running CI. Required conditions to merge
|
User description
What type of PR is this?
Which issue(s) this PR fixes:
issue #23222 #23221
What this PR does / why we need it:
PR Type
Bug fix, Enhancement, Tests, Other
Description
Refactor logging system: Replace string-based logging with structured semantic message keys across multiple packages for better log searchability and filtering
Optimize SQL logging: Abbreviate long SQL strings in logs using
commonutil.Abbreviate()to limit log size and improve readabilityDisable internal operation logging: Add
WithDisableLog()option to SQL execution calls for internal DDL, catalog, schema validation, and vector index operationsConsolidate utility functions: Move duplicate utility functions (
Min,SubStringFromBegin,copyBytes,Abbreviate,CloneBytes) to common package and update all referencesStandardize SQL formatting: Convert SQL keywords to uppercase (SELECT, UPDATE, DELETE, INSERT, DROP, etc.) across multiple files
Fix error handling: Correct variable shadowing in error handling, fix CPU profile return value, and improve error message formatting
Fix configuration: Correct
configStatsGatherIntervaldefault value from 15000 to 150000 millisecondsAdd comprehensive tests: New test files for logtail consumer, error constructors, byte cloning, string abbreviation, and partition state operations
Remove unused code: Delete unused imports, functions, and debug logging statements
Improve test synchronization: Enhance test clarity with better channel naming and buffering in multi-CN bug tests
Diagram Walkthrough
File Walkthrough
15 files
ddl.go
Optimize logging and standardize SQL formatting in DDL operationspkg/sql/compile/ddl.go
c.runSql()calls withc.runSqlWithOptions()to disable loggingfor internal DDL operations
DROP, etc.)
commonutil.Abbreviate()to limit log sizesemantic message keys
logtail_consumer.go
Refactor logging to use structured semantic message keyspkg/vm/engine/disttae/logtail_consumer.go
message keys (e.g.,
logtail.consumer.stopped)logutil.Infof(),logutil.Errorf(),logutil.Warnf()calls touse
logutil.Info(),logutil.Error(),logutil.Warn()with zap fieldsgetLogTailServiceAddr()function logic and improve efficiencylogTagvariablereferences
engine.go
Clean up logging and remove unused imports in enginepkg/vm/engine/disttae/engine.go
github.com/matrixorigin/matrixone/pkg/util/stackfields
Delete()function that was conditional ondatabase name
GetRelationById()functiontable-idinstead oftableId)mo_table_stats.go
Standardize logging format and semantic message keyspkg/vm/engine/disttae/mo_table_stats.go
MO-TABLE-STATS-TASKtomo.table.stats.taskforconsistency
descriptive strings
action,takes,state)pub_sub.go
Disable logging for internal pub/sub catalog operationspkg/sql/compile/pub_sub.go
WithDisableLog()option to SQL execution calls for internalcatalog operations
formatting for readability
runSqlWithResultAndOptions()andrunSqlWithAccountIdAndOptions()methods
build_show.go
Standardize SQL keywords and fix defer orderingpkg/sql/plan/build_show.go
JOIN, CASE, WHEN, THEN, ELSE, END)
defer newStmt.Free()to execute after error check inreturnByRewriteSQL()scope.go
Abbreviate SQL logging and standardize keywordspkg/sql/compile/scope.go
commonutilpackagecommonutil.Abbreviate()upgrade_strategy.go
Disable logging for internal schema validation queriespkg/bootstrap/versions/upgrade_strategy.go
WithDisableLog()option to SQL execution calls inCheckTableColumn(),CheckViewDefinition(), andCheckTableDefinition()sqlexec.go
Disable logging for internal vector index operationspkg/vectorindex/sqlexec/sqlexec.go
WithStatementOption(executor.StatementOption{}.WithDisableLog())to SQL execution options in
RunSql()andRunStreamingSql()functionspartition_state.go
Standardize partition state logging message keyspkg/vm/engine/disttae/logtailreplay/partition_state.go
PS-CREATED)to lowercase with dots (e.g.,
partition.state.created)truncate, etc.)
compile.go
Abbreviate SQL logging and disable internal operation logspkg/sql/compile/compile.go
commonutilpackagecommonutil.Abbreviate()WithDisableLog()option to internal FK detection SQL executioncalls
fuzzyCheck.go
Disable logging for background fuzzy check SQLpkg/sql/compile/fuzzyCheck.go
executorpackagerunSqlWithResultcall torunSqlWithResultAndOptionswithDisableLog()option to suppress logging for background SQL checksbytes.go
Add byte slice cloning utility functionspkg/common/util/bytes.go
CloneBytesIffunction for conditional byte slicecopying
CloneBytesfunction for unconditional byte slice copyingstring.go
Add string abbreviation utility functionpkg/common/util/string.go
Abbreviatefunction to truncate strings withellipsis
string, positive for truncation
error_no_ctx.go
Refactor error creation with format string supportpkg/common/moerr/error_no_ctx.go
errutilpackageNewTxnStaleNoCtxtoNewTxnStaleNoCtxfwith format stringand variadic arguments
errutil.ContextWithNoReportto suppresserror reporting
20 files
error.go
Refactor error constructors for consistency and efficiencypkg/common/moerr/error.go
newError()directlyinstead of intermediate functions
fmt.Sprintf()results to variableserrutil.ContextWithNoReport()wrapper forNewUnsupportedDML()toprevent error reporting
NewDataTruncated()function (replaced byNewDataTruncatedf())mysql_cmd_executor.go
Consolidate utility function imports and usagepkg/frontend/mysql_cmd_executor.go
utiltocommonutilfor claritySubStringFromBegin()calls withcommonutil.Abbreviate()forconsistent string truncation
util.UnsafeBytesToString()withcommonutil.UnsafeBytesToString()util.EncodeUUIDHex()withcommonutil.EncodeUUIDHex()util.go
Remove duplicate utility functions and consolidate importspkg/frontend/util.go
Min()function (use built-in or external alternative)Timeoutstruct and related methods (unused)SubStringFromBegin()function (replaced bycommonutil.Abbreviate())copyBytes()function (replaced bycommonutil.CloneBytesIf())commonutilfor consistencylogStatementStringStatus()function signature for clarityoutput.go
Replace local copyBytes with common utility functionpkg/frontend/output.go
commonutilpackagecopyBytes()calls withcommonutil.CloneBytesIf()for bytearray handling
mysql_protocol_test.go
Use built-in min function instead of custom Minpkg/frontend/mysql_protocol_test.go
Min()function call withmin()built-in functioncatalog.go
Refactor logging with semantic message keyspkg/vm/engine/disttae/cache/catalog.go
timeimportmessage keys (
catalog.cache.update.start.end,catalog.cache.update.start)txn_table.go
Optimize SQL logging and remove debug statementspkg/vm/engine/disttae/txn_table.go
commonUtil.Abbreviate(sql, 500)to truncatelong SQL in logs
mo_columnspartition state(
table.get.snapshot.state.succeed,table.get.snapshot.state.failed)compile2.go
Truncate SQL in logs using common utilitypkg/sql/compile/compile2.go
commonutilpackagecommonutil.Abbreviate(c.sql, 500)in errorand info logs
util.go
Replace custom copyBytes with vector CloneBytesAt methodpkg/cdc/util.go
copyBytes(vec.GetBytesAt(rowIndex))withvec.CloneBytesAt(rowIndex)for JSON typecopyBytes(vec.GetBytesAt(rowIndex))withvec.CloneBytesAt(rowIndex)for string/binary typesbatch_processor.go
Refactor logging with semantic message keyspkg/util/export/batch_processor.go
logger.aggr.start,logger.aggr.handle.count,logger.aggr.bgctx.done,logger.aggr.ctx.done,logger.aggr.exit)zap.Time("end", end)field from aggregation logrecordstocountdata_branch.go
Optimize vector byte operations and comparisonspkg/frontend/data_branch.go
copyBytes(vec.GetBytesAt(...), false)with directvec.GetBytesAt(...)for JSON comparisontypes.CompareValuetobytejson.CompareByteJsoncopyBytescalls with directbytes.Comparefor string/binarytype comparison
server.go
Refactor profile logging with semantic keyspkg/cnservice/server.go
logutil.Errorfto structuredlogutil.Errorwith semantic keys(
profile.save.runtime.failed,profile.writer.close.failed,profile.save.get.ok,profile.save.failed)sql_executor.go
Truncate SQL in logs using common utilitypkg/sql/compile/sql_executor.go
commonutilpackagecommonutil.Abbreviate(sql, 500)in error andinfo logs
server_query.go
Use common utility for byte cloningpkg/cnservice/server_query.go
commonUtilpackagecopyBytesfunction definitioncopyTxnMetato usecommonUtil.CloneBytesinstead of localcopyBytesblocks_iter.go
Remove logging and use error formatting functionpkg/vm/engine/disttae/logtailreplay/blocks_iter.go
fmtimportlogutilimportlogutil.Infofcall and manual error message formatting withmoerr.NewTxnStaleNoCtxffunction callfunc_mo.go
Truncate SQL in subscription loggingpkg/sql/plan/function/func_mo.go
commonutilpackagecommonutil.Abbreviate(sql, 500)insubscription logging
table_scanner.go
Truncate SQL in cleanup loggingpkg/cdc/table_scanner.go
commonutilpackagecommonutil.Abbreviate(sql, 500)in cleanuplogging
cron_task.go
Refactor storage usage logging with semantic keyspkg/util/metric/mometric/cron_task.go
storage_usage.new_accountaccount_id→name,sizeMB→size-mb,snapshotSizeMB→snapshot-mb,createdTime→created-time)mysql_buffer.go
Use built-in min function instead of custom Minpkg/frontend/mysql_buffer.go
Mintomin(lowercase) for built-in minimumfunction
partition.go
Refactor partition truncate logging keypkg/vm/engine/disttae/logtailreplay/partition.go
PS-Truncateto semantic formatpartition.state.truncate8 files
logtail_consumer_test.go
Add comprehensive unit tests for logtail consumerpkg/vm/engine/disttae/logtail_consumer_test.go
send methods
routineControllerwith object poolssubscribe/unsubscribe responses
multi_cn_bug_test.go
Improve test synchronization and error handling claritypkg/queryservice/multi_cn_bug_test.go
connectionErrorOccurredtoconnectionErrorProcessedfor clarityhandling
RequestMultipleCncompletes beforeconnection error is processed
bytes_test.go
Add unit tests for CloneBytesIf utility functionpkg/common/util/bytes_test.go
CloneBytesIf()functionneedCopy=trueand
needCopy=falsecopying
error_test.go
Add comprehensive error constructor test coveragepkg/common/moerr/error_test.go
Test_ForCoverage()to test various errorconstructor functions
(constraint violation, transaction errors, dragonboat errors, TAE
errors)
util_test.go
Update tests to use common utility functionspkg/frontend/util_test.go
commonutilpackageTest_MinMaxtoTest_Maxand removedMintest cases
Test_timeoutfunction entirelyTest_substringFromBeginto usecommonutil.Abbreviateinsteadof local
SubStringFromBeginstring_test.go
Add comprehensive tests for Abbreviate functionpkg/common/util/string_test.go
Abbreviatefunction with comprehensive test casesnegative lengths), and long strings
util_test.go
Update test to use common utility functionpkg/cdc/util_test.go
commonutilpackageTest_copyBytesto usecommonutil.CloneBytesinstead of localcopyBytesfunctionblocks_iter_test.go
Add test for partition state new blocks iteratorpkg/vm/engine/disttae/logtailreplay/blocks_iter_test.go
TestPartitionState_NewBlocksIterto verifyerror handling when snapshot is before partition state start time
3 files
debug.go
Optimize logging and improve CPU profile error handlingcmd/mo-service/debug.go
save.profiles.loop.startedwith structured fields
genCpuProfileto returnnilinstead oferronsuccess
table_space.go
Fix error handling and remove debug loggingpkg/vm/engine/tae/txn/txnimpl/table_space.go
err→err2to avoid shadowingCreateObjectoperationconfig.go
Fix stats gather interval configurationpkg/util/metric/config.go
configStatsGatherIntervaldefault value from15000to150000milliseconds (15 seconds)
2 files
util.go
Format function signaturepkg/vm/engine/disttae/util.go
execReadSqlfunction signature with properindentation
matcher.go
Standardize SQL keyword to uppercasepkg/sql/plan/tools/matcher.go
selectto uppercaseSELECTinparseSqlfunction3 files