Skip to content

feat: persist Prometheus remote write v2 native histograms#8382

Open
shuiyisong wants to merge 18 commits into
GreptimeTeam:mainfrom
shuiyisong:feat/nh
Open

feat: persist Prometheus remote write v2 native histograms#8382
shuiyisong wants to merge 18 commits into
GreptimeTeam:mainfrom
shuiyisong:feat/nh

Conversation

@shuiyisong

@shuiyisong shuiyisong commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This pull request adds write-path persistence for Prometheus remote write v2 native histograms.

  • Decodes PRW v2 native histograms and converts them into write requests.
  • Splits samples and native histograms into separate ContextReqs.
  • Keeps samples on the existing metric-engine/batcher path.
  • Writes native histograms to the same metric table name as one field column: greptime_native_histogram Struct.
  • Lets native histogram rows use the same metric-engine switch and physical-table path as samples when enabled, while bypassing the pending rows batcher.
  • Adds metric-engine support for native histogram logical tables, including Struct field recognition, physical-column alter handling, and sparse/dense batch alignment.
  • Stores common scalar fields, spans, buckets, custom values, and separate int/float count families inside the Struct; the populated family is the histogram type signal.
  • Adds histogram written headers and PROM_STORE_REMOTE_WRITE_HISTOGRAMS.
  • Preserves partial-write accounting for v2 error responses.
  • Adds module docs, unit tests, HTTP handler coverage, binary fixture coverage, and File-backed integration coverage.

Note: This PR only covers write path. We'll add query path in a separate PR.

Native Histogram Data Flow

flowchart TD
    A["HTTP /v1/prometheus/write"] --> B["remote_write_v2"]
    B --> C["decode_remote_write_v2_request"]
    C --> D["into_write_requests"]

    D --> E["samples ContextReq"]
    D --> F["histograms ContextReq"]

    E --> G["write_prometheus_rows_with_progress"]
    G --> H["metric engine / pending batcher when enabled"]

    F --> I["histogram ContextOpt"]
    I --> J["write_prometheus_rows_with_progress"]
    J --> K["same metric-engine flag as samples, no batcher"]
    K --> L["table: <metric>"]

    L --> M["field: greptime_native_histogram Struct"]
    M --> N["struct children: counts, spans, buckets, sum, schema"]

    H --> P["written headers + counters"]
    K --> P
Loading

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@shuiyisong shuiyisong requested a review from sunng87 June 30, 2026 08:34
@shuiyisong shuiyisong requested a review from a team as a code owner June 30, 2026 08:34
@github-actions github-actions Bot added size/L docs-not-required This change does not impact docs. labels Jun 30, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements support for Prometheus Remote Write V2 native histograms, allowing them to be parsed and written to ordinary tables (with a _native_histogram suffix) using list fields for spans and buckets. The feedback focuses on optimizing the high-throughput ingestion path by reducing heap allocations. Specifically, it is recommended to implement a dedicated write_list helper in row_writer.rs to avoid repeated ColumnSchema allocations, pass iterators directly to list field helpers to prevent temporary Vec allocations, and refactor these helpers to accept impl IntoIterator and &str instead of 'static lifetimes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/servers/src/row_writer.rs Outdated
Comment thread src/servers/src/prom_remote_write/v2.rs Outdated
Comment thread src/servers/src/prom_remote_write/v2.rs Outdated
@shuiyisong

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements support for Prometheus Remote Write V2 native histograms, splitting incoming requests into samples and native histograms, which are written to ordinary <metric>_native_histogram tables to accommodate their list schema. It also adds support for writing list-type fields in the row writer, introduces metrics for tracking written histograms, and includes comprehensive tests and documentation. Feedback on the changes suggests a performance optimization in the hot ingestion path to avoid cloning context, table names, and tags when processing sample-only series.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/servers/src/prom_remote_write/v2.rs
Comment thread src/servers/src/prom_remote_write/README.md Outdated
@shuiyisong shuiyisong requested a review from waynexia July 1, 2026 06:17
shuiyisong added 13 commits July 1, 2026 16:34
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
@github-actions github-actions Bot added size/XL and removed size/XXL labels Jul 2, 2026
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
@github-actions github-actions Bot added size/XXL and removed size/XL labels Jul 2, 2026
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
@shuiyisong shuiyisong requested a review from sunng87 July 2, 2026 06:55
@waynexia

waynexia commented Jul 2, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f972f08beb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/servers/src/prom_remote_write/v2.rs Outdated
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required This change does not impact docs. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants