Skip to content

Commit 34c09e1

Browse files
authored
Initial gmp exporter (#342)
Add GMP exporter
1 parent 98a8372 commit 34c09e1

41 files changed

Lines changed: 4184 additions & 125 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

exporter/collector/config.go

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ import (
1818
"fmt"
1919
"time"
2020

21+
"go.opentelemetry.io/collector/model/pdata"
2122
"google.golang.org/api/option"
23+
monitoredrespb "google.golang.org/genproto/googleapis/api/monitoredres"
2224
)
2325

2426
const (
@@ -106,6 +108,18 @@ type MetricConfig struct {
106108
// so we don't send it by default, and don't expose it to users. For some uses, it is
107109
// expected, however.
108110
EnableSumOfSquaredDeviation bool `mapstructure:"sum_of_squared_deviation"`
111+
112+
// GetMetricName is not settable in config files, but can be used by other
113+
// exporters which extend the functionality of this exporter. It allows
114+
// customizing the naming of metrics. baseName already includes type
115+
// suffixes for summary metrics, but does not (yet) include the domain prefix
116+
GetMetricName func(baseName string, metric pdata.Metric) (string, error)
117+
118+
// MapMonitoredResource is not exposed as an option in the configuration, but
119+
// can be used by other exporters to extend the functionality of this
120+
// exporter. It allows overriding the function used to map otel resource to
121+
// monitored resource.
122+
MapMonitoredResource func(pdata.Resource) *monitoredrespb.MonitoredResource
109123
}
110124

111125
type ResourceFilter struct {
@@ -135,6 +149,8 @@ func DefaultConfig() Config {
135149
InstrumentationLibraryLabels: true,
136150
ServiceResourceLabels: true,
137151
CumulativeNormalization: true,
152+
GetMetricName: defaultGetMetricName,
153+
MapMonitoredResource: defaultResourceToMonitoredResource,
138154
},
139155
}
140156
}
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Google Managed Service for Prometheus Collector Exporter
2+
3+
## Building a container image with the googlemanagedprometheus exporter
4+
5+
In your own fork of [open-telemetry/opentelemetry-collector-releases](https://github.com/open-telemetry/opentelemetry-collector-releases), add your own "distribution" directory within the distributions directory, based on either the otelcol or otelcol-contrib distributions. In the `exporters` list in `manifest.yaml`, add:
6+
```yaml
7+
exporters:
8+
- gomod: "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector/googlemanagedprometheus v0.29.0"
9+
```
10+
11+
The syntax of `manifest.yaml` is described in the [Collector Builder documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/54f271b7d473f36b4ecbc21994d59359dbd263f6/cmd/builder/README.md#opentelemetry-collector-builder).
12+
13+
In the `configs` directory, add your collector configuration yaml file, which should look something like:
14+
15+
```yaml
16+
receivers:
17+
prometheus:
18+
config:
19+
scrape_configs:
20+
# TODO: Add your prometheus scrape configuration here.
21+
# Using kubernetes_sd_configs with namespaced resources
22+
# ensures the namespace is set on your metrics.
23+
processors:
24+
batch:
25+
# batch metrics before sending to reduce API usage
26+
send_batch_max_size: 200
27+
send_batch_size: 200
28+
timeout: 5s
29+
memory_limiter:
30+
# drop metrics if memory usage gets too high
31+
check_interval: 1s
32+
limit_percentage: 65
33+
spike_limit_percentage: 20
34+
resourcedetection:
35+
# detect cluster name and location
36+
detectors: [gce, gke]
37+
timeout: 10s
38+
exporters:
39+
googlemanagedprometheus:
40+
```
41+
42+
Change the Dockerfile in your directory within `distributions` to point to your collector config [here](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/Dockerfile#L17).
43+
44+
Finally, build the image:
45+
46+
```sh
47+
DISTRIBUTIONS=my-distribution make build
48+
```
49+
50+
## Additional Options
51+
52+
The [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor) can filter out metrics. The [metricstransformprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor) can manipulate metrics in a variety of ways, including synthesizing new metrics from other metrics, adding or removing labels, renaming metrics, and scaling metrics. `metric_relabl_configs` within the prometheus receiver configuration can also be used to manipulate metrics.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
// Copyright 2022 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// https://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package googlemanagedprometheus
16+
17+
import (
18+
"fmt"
19+
20+
"go.opentelemetry.io/collector/config"
21+
"go.opentelemetry.io/collector/exporter/exporterhelper"
22+
23+
"github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector"
24+
)
25+
26+
// Config defines configuration for Google Cloud Managed Service for Prometheus exporter.
27+
type Config struct {
28+
config.ExporterSettings `mapstructure:",squash"`
29+
GMPConfig `mapstructure:",squash"`
30+
31+
// Timeout for all API calls. If not set, defaults to 12 seconds.
32+
exporterhelper.TimeoutSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
33+
exporterhelper.QueueSettings `mapstructure:"sending_queue"`
34+
exporterhelper.RetrySettings `mapstructure:"retry_on_failure"`
35+
}
36+
37+
// GMPConfig is a subset of the collector config.
38+
type GMPConfig struct {
39+
ProjectID string `mapstructure:"project"`
40+
UserAgent string `mapstructure:"user_agent"`
41+
ClientConfig collector.ClientConfig `mapstructure:",squash"`
42+
}
43+
44+
func (cfg *Config) Validate() error {
45+
if err := cfg.ExporterSettings.Validate(); err != nil {
46+
return fmt.Errorf("exporter settings are invalid :%w", err)
47+
}
48+
if err := collector.ValidateConfig(cfg.toCollectorConfig()); err != nil {
49+
return fmt.Errorf("exporter settings are invalid :%w", err)
50+
}
51+
return nil
52+
}
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
// Copyright 2022 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// https://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package googlemanagedprometheus
16+
17+
import (
18+
"context"
19+
"time"
20+
21+
"go.opentelemetry.io/collector/component"
22+
"go.opentelemetry.io/collector/config"
23+
"go.opentelemetry.io/collector/exporter/exporterhelper"
24+
25+
"github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector"
26+
)
27+
28+
const (
29+
// The value of "type" key in configuration.
30+
typeStr = "googlemanagedprometheus"
31+
defaultTimeout = 12 * time.Second // Consistent with Cloud Monitoring's timeout
32+
)
33+
34+
// NewFactory creates a factory for the googlemanagedprometheus exporter
35+
func NewFactory() component.ExporterFactory {
36+
return component.NewExporterFactory(
37+
typeStr,
38+
createDefaultConfig,
39+
component.WithMetricsExporter(createMetricsExporter),
40+
)
41+
}
42+
43+
// createDefaultConfig creates the default configuration for exporter.
44+
func createDefaultConfig() config.Exporter {
45+
return &Config{
46+
ExporterSettings: config.NewExporterSettings(config.NewComponentID(typeStr)),
47+
TimeoutSettings: exporterhelper.TimeoutSettings{Timeout: defaultTimeout},
48+
RetrySettings: exporterhelper.NewDefaultRetrySettings(),
49+
QueueSettings: exporterhelper.NewDefaultQueueSettings(),
50+
GMPConfig: GMPConfig{
51+
UserAgent: "opentelemetry-collector-contrib/{{version}}",
52+
},
53+
}
54+
}
55+
56+
// createMetricsExporter creates a metrics exporter based on this config.
57+
func createMetricsExporter(
58+
ctx context.Context,
59+
params component.ExporterCreateSettings,
60+
cfg config.Exporter) (component.MetricsExporter, error) {
61+
eCfg := cfg.(*Config)
62+
mExp, err := collector.NewGoogleCloudMetricsExporter(ctx, eCfg.GMPConfig.toCollectorConfig(), params.TelemetrySettings.Logger, params.BuildInfo.Version, eCfg.Timeout)
63+
if err != nil {
64+
return nil, err
65+
}
66+
return exporterhelper.NewMetricsExporter(
67+
cfg,
68+
params,
69+
mExp.PushMetrics,
70+
exporterhelper.WithShutdown(mExp.Shutdown),
71+
// Disable exporterhelper Timeout, since we are using a custom mechanism
72+
// within exporter itself
73+
exporterhelper.WithTimeout(exporterhelper.TimeoutSettings{Timeout: 0}),
74+
exporterhelper.WithQueue(eCfg.QueueSettings),
75+
exporterhelper.WithRetry(eCfg.RetrySettings))
76+
}
77+
78+
func (c *GMPConfig) toCollectorConfig() collector.Config {
79+
// start with whatever the default collector config is.
80+
cfg := collector.DefaultConfig()
81+
// hard-code some config options to make it work with GMP
82+
cfg.MetricConfig.Prefix = "prometheus.googleapis.com"
83+
cfg.MetricConfig.SkipCreateMetricDescriptor = true
84+
cfg.MetricConfig.InstrumentationLibraryLabels = false
85+
cfg.MetricConfig.ServiceResourceLabels = false
86+
// Update metric naming to match GMP conventions
87+
cfg.MetricConfig.GetMetricName = GetMetricName
88+
// Map to the prometheus_target monitored resource
89+
cfg.MetricConfig.MapMonitoredResource = MapToPrometheusTarget
90+
cfg.MetricConfig.EnableSumOfSquaredDeviation = true
91+
// TODO: Change to GMP's method of reset handling.
92+
cfg.MetricConfig.CumulativeNormalization = false
93+
// map the GMP config's fields to the collector config
94+
cfg.ProjectID = c.ProjectID
95+
cfg.UserAgent = c.UserAgent
96+
cfg.MetricConfig.ClientConfig = c.ClientConfig
97+
return cfg
98+
}
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
module github.com/dashpole/opentelemetry-operations-go/exporter/collector/googlemanagedprometheus
2+
3+
go 1.17
4+
5+
require (
6+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector v0.28.0
7+
github.com/stretchr/testify v1.7.1
8+
go.opentelemetry.io/collector v0.49.0
9+
go.opentelemetry.io/collector/model v0.49.0
10+
go.opentelemetry.io/collector/pdata v0.49.0
11+
google.golang.org/genproto v0.0.0-20220405205423-9d709892a2bf
12+
)
13+
14+
require (
15+
cloud.google.com/go v0.100.2 // indirect
16+
cloud.google.com/go/compute v1.5.0 // indirect
17+
cloud.google.com/go/logging v1.4.2 // indirect
18+
cloud.google.com/go/monitoring v1.4.0 // indirect
19+
cloud.google.com/go/trace v1.2.0 // indirect
20+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace v1.5.1 // indirect
21+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.29.1 // indirect
22+
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
23+
github.com/davecgh/go-spew v1.1.1 // indirect
24+
github.com/go-logr/logr v1.2.3 // indirect
25+
github.com/go-logr/stdr v1.2.2 // indirect
26+
github.com/gogo/protobuf v1.3.2 // indirect
27+
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
28+
github.com/golang/protobuf v1.5.2 // indirect
29+
github.com/google/go-cmp v0.5.7 // indirect
30+
github.com/googleapis/gax-go/v2 v2.2.0 // indirect
31+
github.com/knadh/koanf v1.4.1 // indirect
32+
github.com/mitchellh/copystructure v1.2.0 // indirect
33+
github.com/mitchellh/mapstructure v1.4.3 // indirect
34+
github.com/mitchellh/reflectwalk v1.0.2 // indirect
35+
github.com/pmezard/go-difflib v1.0.0 // indirect
36+
go.opencensus.io v0.23.0 // indirect
37+
go.opentelemetry.io/otel v1.6.3 // indirect
38+
go.opentelemetry.io/otel/metric v0.29.0 // indirect
39+
go.opentelemetry.io/otel/sdk v1.6.3 // indirect
40+
go.opentelemetry.io/otel/trace v1.6.3 // indirect
41+
go.uber.org/atomic v1.9.0 // indirect
42+
go.uber.org/multierr v1.8.0 // indirect
43+
go.uber.org/zap v1.21.0 // indirect
44+
golang.org/x/net v0.0.0-20220325170049-de3da57026de // indirect
45+
golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a // indirect
46+
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
47+
golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886 // indirect
48+
golang.org/x/text v0.3.7 // indirect
49+
google.golang.org/api v0.74.0 // indirect
50+
google.golang.org/appengine v1.6.7 // indirect
51+
google.golang.org/grpc v1.45.0 // indirect
52+
google.golang.org/protobuf v1.28.0 // indirect
53+
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
54+
)
55+
56+
replace (
57+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector => ../../collector
58+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace => ../../trace
59+
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping => ../../../internal/resourcemapping
60+
)

0 commit comments

Comments
 (0)