feat/6641435300-save-audit-in-self

[cadem]

Description

Issue(s)

• Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[Copilot]

Pull request overview

This PR adds support for an enterprise audit mode where audit records can be saved inside the audit database itself (instead of being sent to taoskeeper), including plumbing for propagating audit target vnode info, creating the audit supertable on audit DB creation, and adding CI coverage + docs.

Changes:

• Introduces auditSaveInSelf server config and status propagation of audit target SEpSet + vgId.
• Creates the audit supertable (operations) when an audit database is created (with vgroups constrained to 1) and adds vnode-side handling for persisting audit records.
• Adds a new component test for self-audit and updates docs/error codes accordingly.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/ci/cases.task	Adds CI entry for the new self-audit component test.
test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py	New test validating audit records are written into audit.operations when auditSaveInSelf=1.
test/cases/80-Components/01-Taosd/test_com_taosd_audit.py	Adjusts audit DB creation SQL (vgroups=1) and logging.
source/util/src/tjson.c	Adds NULL guard in tjsonGetStringValue2; introduces tjsonGetStringPointer.
include/util/tjson.h	Declares tjsonGetStringPointer with lifetime documentation.
source/util/src/terror.c / include/util/taoserror.h / docs/*error-code.md	Adds and documents new error code for audit DB vgroup restriction.
source/common/src/tglobal.c / include/common/tglobal.h / docs/*taosd.md	Adds and documents auditSaveInSelf config option.
source/common/src/msg/tmsg.c / include/common/tmsg.h	Extends status req/rsp encoding/decoding with audit epSet + vgId; adds SVAuditRecordReq serde helpers.
include/common/tmsgdef.h / source/dnode/mgmt/mgmt_vnode/src/vmHandle.c	Adds vnode msg type TDMT_VND_AUDIT_RECORD and routes it to vnode write queue.
source/dnode/vnode/src/vnd/vnodeSvr.c	Implements vnode-side decoding and persistence path for audit records into audit DB tables.
source/dnode/mnode/impl/src/mndDb.c	Enforces audit DB vgroups == 1; triggers creation of audit supertable when self-save enabled.
source/dnode/mnode/impl/src/mndStb.c / source/dnode/mnode/impl/inc/mndStb.h	Adds audit supertable schema builder + minimal rsp handler wiring.
source/dnode/mnode/impl/src/mndDnode.c / source/dnode/mgmt/mgmt_dnode/src/dmHandle.c	Adds audit epSet/vgId propagation via status messages; masks audit token in logs.
source/dnode/mgmt/mgmt_mnode/src/mmWorker.c / mmHandle.c / mmInt.h / include/common/tmsgcb.h	Adds a dedicated AUDIT_QUEUE and worker routing for audit record responses.
source/dnode/mnode/impl/src/mndTrans.c / mndIndex.c / mndUser.c	Minor logging level adjustment / includes / small formatting.
include/util/tdef.h / include/libs/audit/audit.h	Adds constants (AUDIT_CLIENT_ADD_LEN, AUDIT_STABLE_NAME) and adjusts audit record struct fields.
docs/zh/en/08-operation/16-security.md	Documents self-save mode and the vgroup restriction.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a crucial enhancement to the auditing system by enabling audit information to be stored locally within the database instance itself, bypassing the need for an external taoskeeper service. This provides greater flexibility and control over audit log management. The changes encompass a new configuration parameter, comprehensive documentation updates in both English and Chinese, a new error code to enforce single vgroup constraint for self-auditing databases, and extensive backend modifications to process, store, and retrieve these local audit records. A new test case has also been added to ensure the reliability of this feature.

Highlights

• Local Audit Information Storage: Introduced a new feature allowing audit information to be saved directly within the system (self-auditing) rather than being sent to an external taoskeeper service. This is controlled by the new auditSaveInSelf parameter.
• Configuration and Documentation Updates: Added the auditSaveInSelf parameter to the taosd configuration and updated both English and Chinese documentation to reflect this new functionality and its usage, including the requirement for audit databases to have only one vgroup when this feature is enabled.
• New Error Code: Added a new error code 0x80006108 (TSDB_CODE_AUDIT_DB_NOT_MULTI_VGROUP) to enforce the restriction that audit databases configured for local saving must not have multiple vgroups.
• Internal System Enhancements: Implemented significant internal changes across various modules (common, dnode, mnode, vnode, audit libraries) to support the new local audit mechanism, including new message types, worker queues, serialization logic, and audit super table creation.
• New Test Case: Added a new Python test case (test_com_taosd_self_audit.py) to validate the functionality of the self-auditing feature, covering user creation, super table creation, table creation, data insertion, and deletion.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• docs/en/08-operation/16-security.md
  • Updated documentation to include information about the new auditSaveInSelf parameter and its implications for audit database configuration.
• docs/en/14-reference/01-components/01-taosd.md
  • Added a detailed description for the new auditSaveInSelf configuration parameter.
• docs/en/14-reference/09-error-code.md
  • Added a new error code 0x80006108 for audit databases with multiple vgroups when local saving is enabled.
• docs/zh/08-operation/16-security.md
  • Updated Chinese documentation to include information about the new auditSaveInSelf parameter and its implications for audit database configuration.
• docs/zh/14-reference/01-components/01-taosd.md
  • Added a detailed description for the new auditSaveInSelf configuration parameter in Chinese.
• docs/zh/14-reference/09-error-code.md
  • Added a new error code 0x80006108 for audit databases with multiple vgroups when local saving is enabled in Chinese.
• include/common/tglobal.h
  • Declared the new global boolean variable tsAuditSaveInSelf.
• include/common/tmsg.h
  • Extended SStatusReq and SStatusRsp structs with auditEpSet and auditVgId fields.
  • Defined a new message structure SVAuditRecordReq for audit records.
• include/common/tmsgcb.h
  • Added AUDIT_QUEUE to the EQueueType enumeration.
• include/common/tmsgdef.h
  • Defined a new message type TDMT_VND_AUDIT_RECORD.
• include/libs/audit/audit.h
  • Added mndGetDnodeEpsetByIdFn function pointer type.
  • Increased the size of clientAddress in SAuditRecord to AUDIT_CLIENT_ADD_LEN.
  • Modified the auditRecord function signature to include len and duration parameters.
• include/util/taoserror.h
  • Defined the new error code TSDB_CODE_AUDIT_DB_NOT_MULTI_VGROUP.
• include/util/tdef.h
  • Defined AUDIT_CLIENT_ADD_LEN for client address length.
  • Defined AUDIT_STABLE_NAME for the audit super table name.
• include/util/tjson.h
  • Added tjsonGetStringPointer function to retrieve a pointer to a JSON string value.
• source/common/src/msg/tmsg.c
  • Implemented serialization and deserialization logic for auditEpSet and auditVgId within SStatusReq and SStatusRsp.
  • Implemented serialization and deserialization for the new SVAuditRecordReq message type.
• source/common/src/tglobal.c
  • Initialized tsAuditSaveInSelf to false by default.
  • Added auditSaveInSelf to the server configuration, allowing dynamic modification.
• source/dnode/mgmt/mgmt_dnode/src/dmHandle.c
  • Updated the setAuditDbNameToken function signature to include SEpSet and auditVgId parameters.
  • Added external declarations for getAuditEpSet.
• source/dnode/mgmt/mgmt_mnode/inc/mmInt.h
  • Added auditWorker to the SMnodeMgmt struct.
  • Declared the mmPutMsgToAuditQueue function.
• source/dnode/mgmt/mgmt_mnode/src/mmHandle.c
  • Registered a message handler for TDMT_VND_AUDIT_RECORD_RSP to mmPutMsgToAuditQueue.
• source/dnode/mgmt/mgmt_mnode/src/mmWorker.c
  • Implemented mmPutMsgToAuditQueue to add messages to the audit worker queue.
  • Initialized and started the auditWorker in mmStartWorker and cleaned it up in mmStopWorker.
• source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
  • Registered a message handler for TDMT_VND_AUDIT_RECORD to vmPutMsgToWriteQueue.
• source/dnode/mnode/impl/inc/mndStb.h
  • Declared the mndCreateAuditStb function.
• source/dnode/mnode/impl/src/mndDb.c
  • Added a check during database creation to restrict audit databases to a single vgroup if auditSaveInSelf is enabled.
  • Included logic to call mndCreateAuditStb when an audit database is created with auditSaveInSelf enabled.
• source/dnode/mnode/impl/src/mndDnode.c
  • Modified mndProcessStatusReq to deserialize auditEpSet and auditVgId from status requests.
  • Updated mndProcessStatusReq to serialize auditEpSet and auditVgId into status responses if auditSaveInSelf is enabled.
  • Improved error handling with TAOS_CHECK_GOTO macros.
• source/dnode/mnode/impl/src/mndIndex.c
  • Added an informational log message when adding create index operations to the commit log.
• source/dnode/mnode/impl/src/mndStb.c
  • Added mndProcessAuditRecordRsp to handle audit record responses.
  • Implemented mndBuildAuditStb to define the schema for the audit super table.
  • Implemented mndSetCreateAuditStbRedoActions to set up redo actions for audit super table creation.
  • Implemented mndCreateAuditStb to manage the creation of the audit super table and its associated index.
  • Added informational log messages for commit and redo actions related to super tables.
• source/dnode/mnode/impl/src/mndTrans.c
  • Changed a log level from mInfo to mDebug for transaction hash size.
• source/dnode/mnode/impl/src/mndUser.c
  • Included mndDnode.h and mndVgroup.h headers.
  • Added a call to auditRecord after user creation.
• source/dnode/vnode/src/vnd/vnodeSvr.c
  • Added vnodeProcessAuditRecordReq to handle incoming audit record requests.
  • Implemented helper functions: vnodeDecodeAuditRecord for JSON parsing, vnodeBuildCreateTbReq and vnodePrepareCreateTb for child table creation, vnodePrepareRow for data row preparation, vnodePrepareSubmitTb for submit table data, and vnodeSaveOneAuditRecord for saving individual audit entries.
• source/libs/audit/inc/auditInt.h
  • Added auditEpSet and auditVgId fields to the SAudit internal structure.
• source/util/src/terror.c
  • Defined the error string for TSDB_CODE_AUDIT_DB_NOT_MULTI_VGROUP.
• source/util/src/tjson.c
  • Implemented tjsonGetStringPointer to return a const char* to a JSON string value.
  • Added a null check for pVal in tjsonGetStringValue2.
• test/cases/80-Components/01-Taosd/test_com_taosd_audit.py
  • Modified the existing audit test case to create an audit database with vgroups 1.
• test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py
  • Added a new test file test_com_taosd_self_audit.py to verify the local audit saving functionality.
• test/ci/cases.task
  • Added the new test case test_com_taosd_self_audit.py to the CI task list.

Activity

• The pull request was created by cadem to introduce a new feature for local audit information storage.
• The author included a checklist in the description, indicating that the user manual should be updated, test cases should pass and be automated, and test coverage should not significantly decrease.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new feature to save audit logs locally within a dedicated audit database, instead of sending them to taoskeeper. The changes include a new configuration parameter auditSaveInSelf, a new constraint that audit databases can only have one vgroup, and the necessary logic in mnode and vnode to handle the creation and writing of audit records. The implementation is comprehensive, with updated documentation and new tests. My review focuses on improving documentation clarity, code hygiene, and addressing a potential security risk in logging.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1

• eval() on remoteIP is unsafe even in tests because it executes arbitrary code if the string is compromised or malformed. Prefer parsing as JSON (e.g., json.loads) or using a safer literal parser (e.g., ast.literal_eval) and validating expected keys/types.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• This test uses print() for diagnostic output, while similar tests use tdLog.* (and one was updated in this PR). Use the same logging mechanism here to keep CI logs consistent and to respect any log filtering/formatting the framework provides.
  source/util/src/tjson.c:1
• Parameter validation is incomplete/fragile here: maxLen - 1 can underflow when maxLen <= 0, and pVal is validated only after the length check. It would be safer to validate pVal != NULL and maxLen > 0 before using maxLen in arithmetic, and return TSDB_CODE_INVALID_PARA for invalid arguments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (8)

test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1

• The test operates on db.* objects but this diff doesn’t show db being created (or USE db) anywhere in the test. This makes the test dependent on external state and can fail in clean environments. Create the db database (and/or USE db) explicitly in this test before creating db.stb/db.tb, or switch the statements to a database that the test itself creates.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• The test operates on db.* objects but this diff doesn’t show db being created (or USE db) anywhere in the test. This makes the test dependent on external state and can fail in clean environments. Create the db database (and/or USE db) explicitly in this test before creating db.stb/db.tb, or switch the statements to a database that the test itself creates.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• The test operates on db.* objects but this diff doesn’t show db being created (or USE db) anywhere in the test. This makes the test dependent on external state and can fail in clean environments. Create the db database (and/or USE db) explicitly in this test before creating db.stb/db.tb, or switch the statements to a database that the test itself creates.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• The test operates on db.* objects but this diff doesn’t show db being created (or USE db) anywhere in the test. This makes the test dependent on external state and can fail in clean environments. Create the db database (and/or USE db) explicitly in this test before creating db.stb/db.tb, or switch the statements to a database that the test itself creates.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• This unconditional print will pollute CI output and bypass the test framework’s logging controls. Prefer tdLog.info(...) (or remove the debug output) so logs are consistent and can be filtered/disabled.
  test/cases/80-Components/01-Taosd/test_com_taosd_self_audit.py:1
• Correct the typo “excute” to “execute”.
  source/util/src/tjson.c:1
• pVal is validated only after the length check, which can return TSDB_CODE_OUT_OF_MEMORY even when the real problem is pVal == NULL. Validate pVal before performing size/length checks so the function returns the correct error code deterministically.
  source/util/src/terror.c:1
• The message reads a bit awkwardly (“not allowed to keep multiple vgroups”). Consider rephrasing to something clearer for users, e.g. “Audit database must have exactly 1 vgroup” or “Audit database cannot have multiple vgroups”.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.