Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[exporterbatcher] Update batcher to support serialized bytes based batching. #12641

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[exporterbatcher] Update batcher to support serialized bytes based batching. #12641

wants to merge 2 commits into from

Conversation

sfc-gh-sili
Copy link
Contributor

@sfc-gh-sili sfc-gh-sili commented Mar 16, 2025 •
edited
Loading

Description

This PR

  1. Updates batcher to use Size(sizerType) instead of itemsCount()
  2. For logs, traces, and metrics, it also makes sure that the cachedSize is associated with the sizerType.

This will be followed by https://github.com/open-telemetry/opentelemetry-collector/pull/12635/files which enables turning on this feature from configuration.

Link to tracking issue

#3262

Testing

Documentation

@codecov Codecov
Copy link

codecov bot commented Mar 16, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 83.65385% with 17 lines in your changes missing coverage. Please review.

Project coverage is 91.62%. Comparing base (69908f4) to head (591f645).

Files with missing lines Patch % Lines
...exporterhelper/internal/batcher/default_batcher.go 70.00% 3 Missing ⚠️
exporter/exporterhelper/logs.go 87.50% 3 Missing ⚠️
exporter/exporterhelper/metrics.go 87.50% 3 Missing ⚠️
exporter/exporterhelper/traces.go 87.50% 3 Missing ⚠️
...xporter/exporterhelper/xexporterhelper/profiles.go 57.14% 3 Missing ⚠️
...ter/exporterhelper/internal/requesttest/request.go 0.00% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (83.65%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #12641      +/-   ##
==========================================
- Coverage   91.66%   91.62%   -0.05%     
==========================================
  Files         479      479              
  Lines       26355    26412      +57     
==========================================
+ Hits        24159    24199      +40     
- Misses       1733     1750      +17     
  Partials      463      463

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bogdandrutu
bogdandrutu reviewed Mar 16, 2025
View reviewed changes
exporter/exporterhelper/internal/request/request.go Outdated
@@ -31,6 +33,10 @@ type Request interface {
MergeSplit(context.Context, exporterbatcher.SizeConfig, Request) ([]Request, error)
}

type Sizer interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

bogdandrutu
bogdandrutu reviewed Mar 16, 2025
View reviewed changes
exporter/exporterhelper/logs.go Outdated
@@ -27,14 +28,14 @@ var (
type logsRequest struct {
ld plog.Logs
pusher consumer.ConsumeLogsFunc
cachedSize int
cachedSize map[exporterbatcher.SizerType]int
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to have 2 values than this map which needs heap allocation and lookup is more expensive.

exporter/exporterhelper/logs.go Outdated
case exporterbatcher.SizerTypeBytes:
sz = &sizer.LogsBytesSizer{}
default:
return -1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this design seems a bit off to return -1 here. At least document this behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed -1 to 0 and documented the behavior. Alternative is to panic if the sizer is unknown but I am reluctant to have any panic in the code

@github-actions github-actions bot requested a review from bogdandrutu March 18, 2025 00:22
@sfc-gh-sili sfc-gh-sili force-pushed the sili-fix branch 2 times, most recently from 6526aa3 to 435e943 Compare March 18, 2025 01:22
@sfc-gh-sili sfc-gh-sili changed the title [exporterbatcher] Update batcher to use Size() instead of itemsCount() [exporterbatcher] Update batcher to support serialized bytes based batching. Mar 18, 2025
@sfc-gh-sili sfc-gh-sili mentioned this pull request Mar 19, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mx-psi mx-psi Awaiting requested review from mx-psi mx-psi is a code owner

@dmathieu dmathieu Awaiting requested review from dmathieu dmathieu is a code owner

@dmitryax dmitryax Awaiting requested review from dmitryax dmitryax is a code owner

@bogdandrutu bogdandrutu Awaiting requested review from bogdandrutu bogdandrutu is a code owner

@evan-bradley evan-bradley Awaiting requested review from evan-bradley

@andrzej-stencel andrzej-stencel Awaiting requested review from andrzej-stencel

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sfc-gh-sili @bogdandrutu