feat(keeper): support metrics v2 interface, which is memory mode.

[sheyanjie-qq]

Description

support metrics v2 interface, which is memory mode.

Issue(s)

• Close: https://project.feishu.cn/taosdata_td/feature/detail/6622579928

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the metrics collection and exposure capabilities by introducing a new 'metrics v2' interface. This new approach leverages an in-memory cache to store and serve frequently requested metrics, thereby reducing database overhead and improving response times. It also integrates real-time performance schema queries for critical insights, making the monitoring system more robust and efficient.

Highlights

• New Metrics V2 Interface: Introduced a new /metrics/v2 API endpoint that serves metrics from an in-memory cache and real-time performance schema data, offering a more efficient and responsive metrics collection mechanism.
• In-Memory Metrics Caching: Implemented a MemoryStore to cache incoming metric data, reducing the load on the database for frequently accessed metrics. A MetricCacheMiddleware intercepts and parses metric payloads from specific API paths before storing them.
• Configurable Metric Parsing: Developed a MetricParser to handle different metric report formats (general-metric, cluster-basic, adapter-report) and intelligently filter which tables are cached, with an IncludeTables configuration option to override default exclusions.
• Real-time Performance Schema Metrics: Added a PerformanceCollector to query performance_schema.perf_queries directly, providing real-time insights into slow and very slow queries.
• Gzip Compression: Enabled gzip compression for all API responses to improve network efficiency.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• tools/keeper/api/exporter_test.go
  • Updated NewNodeExporter instantiation to include memoryStore and reporter for v2 metrics.
• tools/keeper/api/metric_middleware.go
  • Added MetricCacheMiddleware to intercept POST requests to specific metric paths, parse their bodies, and store them in an in-memory cache.
  • Implemented shouldCachePath function to determine if a request path should be cached.
• tools/keeper/api/metric_middleware_test.go
  • Added unit tests for MetricCacheMiddleware, covering path matching, request interception, caching behavior, and request body preservation.
• tools/keeper/api/metric_parser.go
  • Added MetricParser to unmarshal various metric JSON payloads (general-metric, cluster-basic, adapter-report) and store them in the MemoryStore.
  • Implemented logic to filter supported tables based on prefixes and an exclusion list, with user-configurable includeTables.
• tools/keeper/api/metric_parser_test.go
  • Added comprehensive unit tests for MetricParser, verifying parsing logic for different metric types, table filtering, and configuration options.
• tools/keeper/api/nodeexporter.go
  • Modified NodeExporter to support a new /metrics/v2 endpoint that serves metrics from the in-memory store and real-time performance schema.
  • Updated NewNodeExporter constructor to accept memoryStore and reporter dependencies.
• tools/keeper/api/performance_collector.go
  • Added PerformanceCollector to query performance_schema.perf_queries for slow and very slow query counts in real-time.
  • Implemented toInt64 helper function for type conversion.
• tools/keeper/api/performance_collector_test.go
  • Added unit tests for PerformanceCollector, verifying its Describe method and the toInt64 helper function.
• tools/keeper/api/v2_integration_test.go
  • Added integration tests for the new metrics v2 interface, covering end-to-end data flow, table filtering, and data expiration scenarios.
• tools/keeper/infrastructure/config/metrics.go
  • Added a new configuration field IncludeTables to Metrics struct, allowing users to specify additional tables for caching in v2 memory mode.
• tools/keeper/process/memory_collector.go
  • Added MemoryStoreCollector as a Prometheus Collector that exposes metrics stored in the MemoryStore, filtering by a configurable maxAge.
• tools/keeper/process/memory_collector_test.go
  • Added unit tests for MemoryStoreCollector, verifying metric collection, max age filtering, and empty store behavior.
• tools/keeper/process/memory_store.go
  • Added MemoryStore for in-memory caching of metric data, including Set, GetAll, GetAllFiltered methods, and a periodic cleanup mechanism.
  • Implemented MetricData struct to hold table name, tags, metrics, and timestamp.
• tools/keeper/process/memory_store_test.go
  • Added unit tests for MemoryStore, covering data storage, updates, filtering, key building, stats retrieval, and concurrent access.
• tools/keeper/system/program.go
  • Integrated the new metrics v2 components: MemoryStore, MetricParser, and MetricCacheMiddleware into the Gin router.
  • Updated NodeExporter initialization to use the new memoryStore and reporter.
  • Enabled gzip compression for the Gin router.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant feature: a v2 metrics interface that uses an in-memory cache for improved performance. The implementation is comprehensive, including data ingestion middleware, a metric parser, an in-memory store, and a new Prometheus collector, all accompanied by extensive tests. My review focuses on enhancing robustness, performance, and configurability. Key suggestions include replacing a fragile time.Sleep with a more reliable synchronization mechanism, making cache settings configurable, and optimizing string handling and type conversions for better performance and safety.

_{Note: Security Review did not run due to the size of the PR.}

[Copilot]

Pull request overview

Adds a new “metrics v2” path that serves Prometheus metrics from an in-memory cache (populated from incoming metric POSTs) and augments it with real-time performance_schema query metrics.

Changes:

• Introduces an in-memory metric store + Prometheus collector to expose cached metrics via /metrics/v2.
• Adds request middleware + parser to synchronously cache selected POST payloads (/general-metric, /taosd-cluster-basic, /adapter_report, etc.), with a config option to re-include default-excluded tables.
• Adds a performance_schema collector to /metrics/v2, plus gzip support in the HTTP server.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tools/keeper/system/program.go	Enables gzip and wires up v2 memory cache + middleware; updates NodeExporter construction.
tools/keeper/system/program.go	Adds global gzip middleware and v2 memory-mode initialization.
tools/keeper/process/memory_store.go	New in-memory metric storage with TTL cleanup and stats.
tools/keeper/process/memory_store_test.go	Unit tests for MemoryStore behavior.
tools/keeper/process/memory_collector.go	New Prometheus collector that emits metrics from MemoryStore.
tools/keeper/process/memory_collector_test.go	Tests for MemoryStoreCollector behavior.
tools/keeper/infrastructure/config/metrics.go	Adds metrics.includeTables config/env/flag for v2 cache inclusion.
tools/keeper/api/metric_middleware.go	New Gin middleware to intercept metric POSTs and cache them.
tools/keeper/api/metric_middleware_test.go	Tests for caching middleware path/method behavior.
tools/keeper/api/metric_parser.go	New parser that filters tables and stores metrics into MemoryStore.
tools/keeper/api/metric_parser_test.go	Parser tests for multiple endpoints and include/exclude behavior.
tools/keeper/api/nodeexporter.go	Adds /metrics/v2 endpoint and registers memory + perf collectors.
tools/keeper/api/performance_collector.go	New collector querying performance_schema.perf_queries for slow-query counts.
tools/keeper/api/performance_collector_test.go	Tests for performance collector Describe and toInt64.
tools/keeper/api/v2_integration_test.go	End-to-end tests for v2: cache -> collector -> promhttp output.
tools/keeper/api/exporter_test.go	Updates NodeExporter constructor usage for new v2 dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 12 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.