-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Rough sketch (draft 3) limiter extension and middleware #12700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 9 commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
6aac7ae
config and limiter extension
jmacd 2a1818d
config and middleware extension
jmacd b519182
add limiter skeleton
jmacd 4434dac
add request limiter calls
jmacd 25a385c
At the two locations where I have written @@@, I realize that I have …
jmacd 8fe3941
split rate limiter, resource limiter
jmacd 84838fc
todos
jmacd cdc6698
skeleton ratelimiterextension
jmacd 88b203b
rate limiter outline
jmacd b602b18
HTTP client and server network bytes
jmacd cec3306
add limiter provider
jmacd 060c756
Follow 17002
jmacd ee8dd47
Let rate limiter accept request resource-weighted limits
jmacd 9369495
Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…
jmacd af51711
middleware config: more direct, extension naming more consistent
jmacd 2c3a406
sketch grpc middleware
jmacd 8ca49f0
extend confighttp and configgrpc
jmacd 1fa405e
memory limiter
jmacd e125898
multi limiter provider
jmacd a6e6b17
OTLP receiver limiter
jmacd 10cfbb3
add direct grpc limiter support (this could be a helper)
jmacd 1f4a724
add direct http limiter support (this could be a helper)
jmacd ee0c2e0
merge
jmacd 6d194a8
factor limiterhelper
jmacd 9f66533
remove/rename
jmacd 0e30a3a
Use non-nil ReleaseFunc everywhere
jmacd 70296f0
Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…
jmacd a79da5f
Merge branch 'main' of github.com:jmacd/opentelemetry-collector into …
jmacd 854108f
update from main
jmacd fb5b8b9
Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…
jmacd fc7ba55
hide the option and config, eliminate the limiterhelper.Consumer
jmacd ea8152a
wip
jmacd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include ../../Makefile.Common |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| // Package configlimiter implements the configuration settings to | ||
| // apply rate limiting on incoming requests, and allows | ||
| // components to configure rate limiting behavior. | ||
| package configlimiter // import "go.opentelemetry.io/collector/config/configlimiter" | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
|
|
||
| "go.opentelemetry.io/collector/component" | ||
| "go.opentelemetry.io/collector/extension/extensionlimiter" | ||
| ) | ||
|
|
||
| var ( | ||
| errLimiterNotFound = errors.New("limiter not found") | ||
| errNotLimiter = errors.New("requested extension is not a limiter") | ||
| ) | ||
|
|
||
| // Limiter defines the rate limiting settings for a component. | ||
| type Limiter struct { | ||
| // LimiterID specifies the name of the extension to use in order to apply rate limiting. | ||
| LimiterID component.ID `mapstructure:"limiter,omitempty"` | ||
| } | ||
|
|
||
| // GetLimiter attempts to select the appropriate extensionlimiter.Limiter from the list of extensions, | ||
| // based on the requested extension name. If a limiter is not found, an error is returned. | ||
| func (l Limiter) GetRateLimiter(_ context.Context, extensions map[component.ID]component.Component) (extensionlimiter.RateLimiter, error) { | ||
| if ext, found := extensions[l.LimiterID]; found { | ||
| if limiter, ok := ext.(extensionlimiter.RateLimiter); ok { | ||
| return limiter, nil | ||
| } | ||
| return nil, errNotLimiter | ||
| } | ||
|
|
||
| return nil, fmt.Errorf("failed to resolve limiter %q: %w", l.LimiterID, errLimiterNotFound) | ||
| } | ||
|
|
||
| // GetResourceLimiter attempts to select the appropriate extensionlimiter.ResourceLimiter from the list of extensions, | ||
| // based on the requested extension name. If a limiter is not found, an error is returned. | ||
| func (l Limiter) GetResourceLimiter(_ context.Context, extensions map[component.ID]component.Component) (extensionlimiter.ResourceLimiter, error) { | ||
| if ext, found := extensions[l.LimiterID]; found { | ||
| if limiter, ok := ext.(extensionlimiter.ResourceLimiter); ok { | ||
| return limiter, nil | ||
| } | ||
| return nil, errNotLimiter | ||
| } | ||
|
|
||
| return nil, fmt.Errorf("failed to resolve resource limiter %q: %w", l.LimiterID, errLimiterNotFound) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| module go.opentelemetry.io/collector/config/configlimiterg | ||
|
|
||
| go 1.23.0 | ||
|
|
||
| require ( | ||
| go.opentelemetry.io/collector/component v1.28.1 | ||
| go.opentelemetry.io/collector/extension/extensionlimiter v0.0.0-00010101000000-000000000000 | ||
| ) | ||
|
|
||
| require ( | ||
| github.com/gogo/protobuf v1.3.2 // indirect | ||
| go.opentelemetry.io/collector/pdata v1.28.1 // indirect | ||
| go.opentelemetry.io/otel v1.35.0 // indirect | ||
| go.opentelemetry.io/otel/metric v1.35.0 // indirect | ||
| go.opentelemetry.io/otel/sdk/metric v1.35.0 // indirect | ||
| go.opentelemetry.io/otel/trace v1.35.0 // indirect | ||
| go.uber.org/multierr v1.11.0 // indirect | ||
| go.uber.org/zap v1.27.0 // indirect | ||
| golang.org/x/net v0.37.0 // indirect | ||
| golang.org/x/sys v0.31.0 // indirect | ||
| golang.org/x/text v0.23.0 // indirect | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect | ||
| google.golang.org/grpc v1.71.0 // indirect | ||
| google.golang.org/protobuf v1.36.5 // indirect | ||
| ) | ||
|
|
||
| replace go.opentelemetry.io/collector/component => ../../component | ||
|
|
||
| replace go.opentelemetry.io/collector/extension/extensionlimiter => ../../extension/extensionlimiter |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include ../../Makefile.Common |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| // Package configmiddleware implements the configuration settings to | ||
| // apply middleware processing on telemetry data, and allows | ||
| // components to configure middleware behavior. | ||
| package configmiddleware // import "go.opentelemetry.io/collector/config/configmiddleware" | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
|
|
||
| "go.opentelemetry.io/collector/component" | ||
| "go.opentelemetry.io/collector/extension/extensionmiddleware" | ||
| ) | ||
|
|
||
| var ( | ||
| errMiddlewareNotFound = errors.New("middleware not found") | ||
| errNotMiddleware = errors.New("requested extension is not a middleware") | ||
| errNotHTTPServer = errors.New("requested extension is not an HTTP server middleware") | ||
| errNotGRPCServer = errors.New("requested extension is not a gRPC server middleware") | ||
| errNotHTTPClient = errors.New("requested extension is not an HTTP client middleware") | ||
| errNotGRPCClient = errors.New("requested extension is not a gRPC client middleware") | ||
| ) | ||
|
|
||
| // Middleware defines the middleware settings for a component. | ||
| type Middleware struct { | ||
| // MiddlewareID specifies the name of the extension to use in order to apply middleware processing. | ||
| MiddlewareID component.ID `mapstructure:"middleware,omitempty"` | ||
| } | ||
|
|
||
| // GetHTTPServerMiddleware attempts to select the appropriate extensionmiddleware.HTTPServer from the list of extensions, | ||
| // based on the requested extension name. If a middleware is not found, an error is returned. | ||
| func (m Middleware) GetHTTPServerMiddleware(_ context.Context, extensions map[component.ID]component.Component) (extensionmiddleware.HTTPServer, error) { | ||
| if ext, found := extensions[m.MiddlewareID]; found { | ||
| if server, ok := ext.(extensionmiddleware.HTTPServer); ok { | ||
| return server, nil | ||
| } | ||
| return nil, errNotHTTPServer | ||
| } | ||
|
|
||
| return nil, fmt.Errorf("failed to resolve middleware %q: %w", m.MiddlewareID, errMiddlewareNotFound) | ||
| } | ||
|
|
||
| // GetGRPCServerMiddleware attempts to select the appropriate extensionmiddleware.GRPCServer from the list of extensions, | ||
| // based on the requested extension name. If a middleware is not found, an error is returned. | ||
| func (m Middleware) GetGRPCServerMiddleware(_ context.Context, extensions map[component.ID]component.Component) (extensionmiddleware.GRPCServer, error) { | ||
| if ext, found := extensions[m.MiddlewareID]; found { | ||
| if server, ok := ext.(extensionmiddleware.GRPCServer); ok { | ||
| return server, nil | ||
| } | ||
| return nil, errNotGRPCServer | ||
| } | ||
|
|
||
| return nil, fmt.Errorf("failed to resolve middleware %q: %w", m.MiddlewareID, errMiddlewareNotFound) | ||
| } | ||
|
|
||
| // GetHTTPClientMiddleware attempts to select the appropriate extensionmiddleware.HTTPClient from the list of extensions, | ||
| // based on the component id of the extension. If a middleware is not found, an error is returned. | ||
| // This should be only used by HTTP clients. | ||
| func (m Middleware) GetHTTPClientMiddleware(_ context.Context, extensions map[component.ID]component.Component) (extensionmiddleware.HTTPClient, error) { | ||
| if ext, found := extensions[m.MiddlewareID]; found { | ||
| if client, ok := ext.(extensionmiddleware.HTTPClient); ok { | ||
| return client, nil | ||
| } | ||
| return nil, errNotHTTPClient | ||
| } | ||
| return nil, fmt.Errorf("failed to resolve middleware %q: %w", m.MiddlewareID, errMiddlewareNotFound) | ||
| } | ||
|
|
||
| // GetGRPCClientMiddleware attempts to select the appropriate extensionmiddleware.GRPCClient from the list of extensions, | ||
| // based on the component id of the extension. If a middleware is not found, an error is returned. | ||
| // This should be only used by gRPC clients. | ||
| func (m Middleware) GetGRPCClientMiddleware(_ context.Context, extensions map[component.ID]component.Component) (extensionmiddleware.GRPCClient, error) { | ||
| if ext, found := extensions[m.MiddlewareID]; found { | ||
| if client, ok := ext.(extensionmiddleware.GRPCClient); ok { | ||
| return client, nil | ||
| } | ||
| return nil, errNotGRPCClient | ||
| } | ||
| return nil, fmt.Errorf("failed to resolve middleware %q: %w", m.MiddlewareID, errMiddlewareNotFound) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| module go.opentelemetry.io/collector/config/configlimiter | ||
|
|
||
| go 1.23.0 | ||
|
|
||
| replace go.opentelemetry.io/collector/component => ../../component | ||
|
|
||
| replace go.opentelemetry.io/collector/extension/extensionmiddleware => ../../extension/extensionmiddleware | ||
|
|
||
| //replace go.opentelemetry.io/collector/extension/extensionmiddleware/extensionmiddlewaretest => ../../extension/extensionmiddleware/extensionmiddlewaretest | ||
|
|
||
| require ( | ||
| go.opentelemetry.io/collector/component v0.0.0-00010101000000-000000000000 | ||
| go.opentelemetry.io/collector/extension/extensionmiddleware v0.0.0-00010101000000-000000000000 | ||
| ) | ||
|
|
||
| require ( | ||
| github.com/gogo/protobuf v1.3.2 // indirect | ||
| go.opentelemetry.io/collector/pdata v1.28.1 // indirect | ||
| go.opentelemetry.io/otel v1.35.0 // indirect | ||
| go.opentelemetry.io/otel/metric v1.35.0 // indirect | ||
| go.opentelemetry.io/otel/trace v1.35.0 // indirect | ||
| go.uber.org/multierr v1.11.0 // indirect | ||
| go.uber.org/zap v1.27.0 // indirect | ||
| golang.org/x/net v0.37.0 // indirect | ||
| golang.org/x/sys v0.31.0 // indirect | ||
| golang.org/x/text v0.23.0 // indirect | ||
| google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect | ||
| google.golang.org/grpc v1.71.0 // indirect | ||
| google.golang.org/protobuf v1.36.5 // indirect | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.