feat: add time series statistics profile #17809
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
pr-feature
dqhl76
force-pushed
the
time-series-profile
branch
2 times, most recently
from
971528e to
dfdef76
Compare
dqhl76
added
ci-benchmark
Docker Image for PR
|
|
Disabled in log by default. Enable by adding |
dqhl76
force-pushed
the
time-series-profile
branch
from
3cf1865 to
a0ecd35
Compare
zhang2014
reviewed
Apr 21, 2025
zhang2014
reviewed
Apr 21, 2025
Docker Image for PR
|
|
@zhang2014 Benchmark looks good. Please take a look again |
zhang2014
reviewed
Apr 24, 2025
zhang2014
reviewed
Apr 24, 2025
zhang2014
reviewed
Apr 24, 2025
zhang2014
reviewed
Apr 24, 2025
zhang2014
reviewed
Apr 24, 2025
zhang2014
reviewed
Apr 24, 2025
dqhl76
force-pushed
the
time-series-profile
branch
from
5afa43f to
2ae2c1c
Compare
zhang2014
reviewed
Apr 30, 2025
zhang2014
approved these changes
May 6, 2025
dqhl76
added
ci-benchmark
This reverts commit 6c17ab4.
dqhl76
added
ci-benchmark
Docker Image for PR
|
Docker Image for PR
|
zhang2014
added
ci-benchmark
Docker Image for PR
|
|
Performance degradation occurs in the range of: https://github.com/databendlabs/databend/pull/17809/files/c2d2f4f5b303981f930efca7ec72325ce833cf86..53ccd2d58316722c3661750bc68f199e4ecaf0e2 I will investigate the cause later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
close: #17779
Add time series statistics profile. Iniaitily, we support output_rows and output_bytes. This allows us to visualize and compare the data processing speed of different plan nodes.
Disabled in log by default. Enable by adding
databend::log::time_series=INFO{ "query_id": "76f9fa48-7843-456b-9519-f53daae9a1d7", "plans": [ { "plan_id": 0, "data": [ [[ 1745932982,5 ]], [[ 1745932982,40 ]] ] }, { "plan_id": 1, "data": [ [[ 1745932982,5 ]], [[ 1745932982,40 ]] ] }, { "plan_id": 2, "data": [ [[ 1745932980,20512768,55312384,24174848 ]], [[ 1745932980,164102144,442499072,193398784 ]] ] } ], "desc": [ { "name": "OutputRows", "index": 0 }, { "name": "OutputBytes", "index": 1 } ] }Output Format
We use a compressed representation for time series data points to save space and improve transmission efficiency. Instead of storing each (timestamp, value) pair separately, consecutive data points with incrementing timestamps are grouped into segments. Each segment is represented as a list where the first element is the starting timestamp, and the subsequent elements are the values for each consecutive timestamp in that segment.
For example, the raw data
[(t0, v0), (t1, v1), (t2, v2), (t4, v4), (t5, v5)]will be compressed as
[[t0, v0, v1, v2], [t4, v4, v5]].The timestamp is the unix timestamp in second
Tests
Type of change
This change is