feat(keeper): migrate from main to 3.3.8 and add metrics v2 support#34698
Open
sheyanjie-qq wants to merge 2 commits into3.3.8from
Open
feat(keeper): migrate from main to 3.3.8 and add metrics v2 support#34698sheyanjie-qq wants to merge 2 commits into3.3.8from
sheyanjie-qq wants to merge 2 commits into3.3.8from
Conversation
- Add metrics v2 support with in-memory cache mode - Add prometheus config for cache management - Refactor config into modular files - Keep audit implementation from 3.3.8 for compatibility - Update dependencies and fix bugs Close: https://project.feishu.cn/taosdata_td/feature/detail/6622579928
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the keeper's metrics infrastructure by introducing a new Prometheus v2 API that utilizes an in-memory caching mechanism. This change aims to provide more immediate and performant access to metrics data, reducing reliance on direct database queries for frequently accessed information. Concurrently, the configuration system has been reorganized into a modular structure, enhancing clarity and ease of management. These updates, alongside critical dependency upgrades and bug fixes, contribute to a more modern, efficient, and secure monitoring solution. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates taoskeeper toward the 3.3.8 baseline while adding Prometheus metrics v2 support via an in-memory cache (with TTL-based cleanup), plus modularized configuration (Prometheus/SSL/TDengine/env/log).
Changes:
- Add in-memory metric cache pipeline (middleware + parser + store) and expose
/metrics/v2with optionalmax_age. - Introduce Prometheus/SSL/TDengine/env modular config + TOML updates.
- Extend DB connector to support bearer token authentication and update related tests/deps.
Reviewed changes
Copilot reviewed 35 out of 37 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/keeper/system/program_test.go | Adds HTTP/HTTPS start tests and adjusts init cleanup behavior. |
| tools/keeper/system/program.go | Wires v2 memory cache + middleware, adds HTTPS support, and shuts down cache on stop. |
| tools/keeper/process/memory_store_test.go | Adds unit tests for in-memory TTL cache behavior and concurrency. |
| tools/keeper/process/memory_store.go | Implements in-memory metric store with TTL cleanup loop and stats. |
| tools/keeper/process/memory_collector_test.go | Adds tests for Prometheus collector emitting cached metrics. |
| tools/keeper/process/memory_collector.go | Prometheus collector for exporting cached metrics with max-age filtering. |
| tools/keeper/process/builder_test.go | Updates config type rename in tests. |
| tools/keeper/process/builder.go | Updates config type rename for metrics expansion. |
| tools/keeper/main.go | Minor variable rename for clarity. |
| tools/keeper/infrastructure/config/tdengine.go | Extracts TDengine config initialization into a module. |
| tools/keeper/infrastructure/config/ssl_test.go | Adds SSL config parsing/env tests. |
| tools/keeper/infrastructure/config/ssl.go | Adds SSL config module for HTTPS serving. |
| tools/keeper/infrastructure/config/prometheus.go | Adds Prometheus v2 cache config (includeTables/cacheTTL). |
| tools/keeper/infrastructure/config/metrics.go | Refactors metrics config and registers flags/env defaults. |
| tools/keeper/infrastructure/config/metric_test.go | Minor comment formatting changes in test helper. |
| tools/keeper/infrastructure/config/log.go | Refactors log config initialization into a module. |
| tools/keeper/infrastructure/config/env.go | Extracts environment config initialization into a module. |
| tools/keeper/infrastructure/config/config.go | Integrates new modular config structs (Metrics/Prometheus/SSL) and init calls. |
| tools/keeper/go.sum | Updates dependency checksums for new/updated modules. |
| tools/keeper/go.mod | Updates Go version and dependencies (service/logrus/driver-go, adds gomonkey). |
| tools/keeper/db/connector_test.go | Adds enterprise token-connector tests and reorganizes special-char tests. |
| tools/keeper/db/connector.go | Adds NewConnectorWithDbAndToken* and bearerToken DSN parameter. |
| tools/keeper/config/taoskeeper_enterprise.toml | Adds [prometheus] v2 cache configuration block. |
| tools/keeper/config/taoskeeper.toml | Adds [prometheus] v2 cache configuration block. |
| tools/keeper/cmd/command.go | Quotes role field in SQL to avoid reserved-word conflicts. |
| tools/keeper/api/v2_integration_test.go | Adds end-to-end v2 cache → collector → HTTP output tests. |
| tools/keeper/api/tables.go | Quotes role column in table DDL. |
| tools/keeper/api/report.go | Adds non-forever-retry DB connect helper for collectors. |
| tools/keeper/api/performance_collector_test.go | Adds unit tests for performance collector and toInt64. |
| tools/keeper/api/performance_collector.go | Adds performance_schema real-time Prometheus collector. |
| tools/keeper/api/nodeexporter.go | Adds /metrics/v2 endpoint and registers memory+performance collectors. |
| tools/keeper/api/metric_parser_test.go | Adds tests for parsing/caching metrics for v2 mode. |
| tools/keeper/api/metric_parser.go | Implements parser for caching selected endpoints into MemoryStore. |
| tools/keeper/api/metric_middleware_test.go | Adds middleware tests ensuring caching + body preservation. |
| tools/keeper/api/metric_middleware.go | Adds middleware to intercept metric POST bodies and cache synchronously. |
| tools/keeper/api/exporter_test.go | Updates NodeExporter construction to include memory store + reporter. |
| tools/keeper/api/adapter2_test.go | Updates metrics config type rename in adapter tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a significant new feature: metrics v2 with an in-memory cache, which improves performance by avoiding database queries for metrics. However, the implementation contains several Denial of Service (DoS) vulnerabilities, including a potential process panic due to lack of validation for metric names used in Prometheus descriptors, and memory exhaustion risks due to unlimited request body reading and an unbounded in-memory store on unauthenticated endpoints. Beyond these critical security concerns, the review also focuses on improving the robustness and maintainability of the new components, despite the positive refactoring of configuration and dependency updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.