Create a SizedGenerator for use in OTel metrics#1355
Merged
Conversation
This was referenced May 13, 2025
Collaborator
Author
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
e4db940 to
e5b7497
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
2 times, most recently
from
87fa48b to
f24978e
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
e5b7497 to
9fbe654
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
f24978e to
e954798
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
9fbe654 to
ce2d61a
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
e954798 to
c631da0
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
2 times, most recently
from
1712dab to
96508d5
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
2 times, most recently
from
8dc968c to
04b3096
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
96508d5 to
c1d221c
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
04b3096 to
c0ed5ff
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
2 times, most recently
from
287106d to
0e2b5e0
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
2 times, most recently
from
1070aa3 to
30286ee
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
0e2b5e0 to
b240441
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
30286ee to
8f095f4
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
b240441 to
9fac6a2
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
8f095f4 to
9a1e8e2
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
9fac6a2 to
97b5d15
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
from
9a1e8e2 to
585b425
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
97b5d15 to
a586e7e
Compare
blt
force-pushed
the
blt/otel_metric_templates_are_direct_protobuf_types
branch
2 times, most recently
from
e3a8386 to
867ab2b
Compare
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
19b629a to
fb7a2b1
Compare
blt
changed the base branch from
blt/otel_metric_templates_are_direct_protobuf_types
to
graphite-base/1355
This commit attempts to address the payload generation issues -- blocks are too big to be useful, depending on user configuration -- by making the Generator aware of the byte budget. This is still failing tests as of this commit but it compiles. REF SMPTNG-659 Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
fb7a2b1 to
62c1fbf
Compare
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
blt
force-pushed
the
blt/create_a_sizedgenerator_for_use_in_otel_metrics
branch
from
62c1fbf to
7fe5e85
Compare
scottopell
approved these changes
May 16, 2025
| use super::templates::GeneratorError; | ||
| use crate::{Error, Generator, common::config::ConfRange, common::strings::Pool}; | ||
| use opentelemetry_proto::tonic::common::v1::{AnyValue, KeyValue, any_value}; | ||
| use prost::Message; |
Contributor
There was a problem hiding this comment.
I think you can remove this since you import in the test module
| pub(crate) fn fetch<R>( | ||
| &mut self, | ||
| rng: &mut R, | ||
| budget: &mut usize, |
Contributor
There was a problem hiding this comment.
small suggestion for extra clarity
Suggested change
| budget: &mut usize, | |
| encoded_budget: &mut usize, |
|
|
||
| // Generate new instances until either context_cap is hit or the | ||
| // remaining space drops below our lookup interval. | ||
| if self.len < self.context_cap { |
Contributor
There was a problem hiding this comment.
the goal here is to "lazily" generate new contexts when needed rather than doing it entirely up front, and it seems like part of that motivation is so that we can call generator.generate() with the "updated" limit and ensure that the templates are as useful as possible
| } | ||
| } | ||
|
|
||
| fn cut_data_points(metric: Metric) -> Metric { |
Contributor
There was a problem hiding this comment.
looks like this may have some allocations involved, could this take a mutable reference and modify it in-place?
or if the lifetimes are hard, maybe use std::mem::take to steal the memory?
| /// Generate a new instance of `Self::Output`. Implementations MUST uphold | ||
| /// the following properties: | ||
| /// | ||
| /// * `budget` is decremented if and only if return is Ok |
Contributor
There was a problem hiding this comment.
Non-blocking comment, but a suggestion for this API would be to return a tuple of Self::Output (the generated item) and usize (encoded size of generated item) and push this invariant up to the caller to do the accounting. Would slightly simplify the implementations of generate (which are already plenty complex)
Collaborator
Author
This was referenced May 17, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This commit attempts to address the payload generation issues -- blocks
are too big to be useful, depending on user configuration -- by making the
Generator aware of the byte budget. This is still failing tests as of this
commit but it compiles.
Motivation
REF SMPTNG-659