Add support for push-gateway and OTLP (#67)

Forced to update opentelemetry-go as well as there was no way to bring
the update without also going into otel breaking changes.

Author: gagbo

.github/workflows/build.yml

@@ -7,7 +7,7 @@ jobs:
 
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Go
         uses: actions/setup-go@v3

.github/workflows/coverage.yml

@@ -10,7 +10,7 @@ jobs:
     continue-on-error: true
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         fetch-depth: 10
 

.github/workflows/release.yml

@@ -29,7 +29,7 @@ jobs:
             goarch: arm64
             ext: ''
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           # We need all tags
           fetch-depth: 0

CHANGELOG.md

@@ -10,8 +10,35 @@ versioning](https://go.dev/doc/modules/version-numbers).
 
 ### Added
 
+- [Prometheus collector] `Init` call now takes an optional `PushConfiguration`
+  argument that allows to push metrics to PushGateway, _on top_ of exposing the
+  metrics to an endpoint.
+  Using the push gateway is very useful to automatically send metrics when a
+  web service is autoscaled and has replicas being created or deleted during
+  normal operations.
+  + the `CollectorURL` in the `PushConfiguration` is mandatory and follows the
+    API of `prometheus/push.New`
+  + the `JobName` parameter is optional, and `autometrics` will do a best effort
+    to fill the value.
+
+- [OpenTelemetry collector] `Init` call now takes an optional `PushConfiguration`
+  argument that allows to push metrics in OTLP format, _instead_ of exposing the
+  metrics to a Promtheus-format endpoint.
+  Using an OTLP collector is very useful to automatically send metrics when a
+  web service is autoscaled and has replicas being created or deleted during
+  normal operations.
+  + the `CollectorURL` in the `PushConfiguration` is mandatory
+  + the `JobName` parameter is optional, and `autometrics` will do a best effort
+    to fill the value.
+
 ### Changed
 
+- [All] `autometrics.Init` function (for both `prometheus` and `otel` variants)
+  take an extra optional argument. See the `Added` section above for details.
+- [All] `autometrics.Init` function (for both `prometheus` and `otel` variants)
+  now return a [CancelCauseFunc](https://pkg.go.dev/context#CancelCauseFunc)
+  to cleanly shutdown (early or as a normal shutdown process) all metric collection.
+
 ### Deprecated
 
 ### Removed

README.md

@@ -93,11 +93,16 @@ import (
 And then in your main function initialize the metrics
 
 ``` go
-	autometrics.Init(
+	shutdown, err := autometrics.Init(
 		nil,
 		autometrics.DefBuckets,
 		autometrics.BuildInfo{Version: "0.4.0", Commit: "anySHA", Branch: "", Service: "myApp"},
+		nil,
 	)
+	if err != nil {
+		log.Fatalf("could not initialize autometrics: %s", err)
+	}
+	defer shutdown(nil)
 ```
 
 Everything in `BuildInfo` is optional. It will add relevant information on the
@@ -237,10 +242,11 @@ import (
 
 
 func main() {
-	autometrics.Init(
+	shutdown, err := autometrics.Init(
 		nil,
 		autometrics.DefBuckets,
 		autometrics.BuildInfo{Version: "0.4.0", Commit: "anySHA", Branch: "", Service: "myApp"},
+		nil,
 	)
 	http.Handle("/metrics", promhttp.Handler())
 }
@@ -346,11 +352,12 @@ the `Init` function takes a meter name for the `otel_scope` label of the exporte
 metric. You can use the name of the application or its version for example
 
 ``` patch
-	autometrics.Init(
+	shutdown, err := autometrics.Init(
 -		nil,
 +		"myApp/v2/prod",
 		autometrics.DefBuckets,
 		autometrics.BuildInfo{ Version: "2.1.37", Commit: "anySHA", Branch: "", Service: "myApp" },
+		nil,
 	)
 ```
 
@@ -361,6 +368,68 @@ metric. You can use the name of the application or its version for example
 +//go:generate autometrics --otel
 ```
 
+#### Push-based workflows
+
+<details>
+<summary><i>Why would I use a push-based workflow?</i></summary>
+
+If you have an auto-scaled service (with instances spinning up and down),
+maintaining the configuration/discovery of instances on the Prometheus side of
+things can be hard. Using a push-based workflow inverts the burden of
+configuration: all your instances generate a specific ID, and they just need to
+push metrics to a given URL. So the main advantages of a push-based workflow
+appear when the the set of machines producing metrics is dynamic:
+
+- Your Prometheus configuration does not need to be dynamic anymore, it's "set
+  and forget" again
+- No need to configure service discovery separately (which can be error-prone)
+
+
+It can be summarized with one sentence. <b>The monitoring stack
+(Prometheus/OpenTelemetry collector) does not need to know the infrastructure of
+application deployment; nor does the application code need to know the
+infrastructure of the monitoring stack. Decoupling prevents
+configuration-rot.</b>
+
+</details>
+
+If you don't want to/cannot configure your Prometheus instance to scrape the
+instrumented code, Autometrics provides a way to push metrics instead of relying
+on a polling collection process.
+
+> **Note**
+> It is strongly advised to use the OpenTelemetry variant of Autometrics to support push-based metric
+collection. Prometheus push gateways make aggregation of data across multiple sources harder.
+
+_How can I use a push-based workflow with Autometrics?_
+
+If you have a Prometheus [push
+gateway](https://prometheus.io/docs/instrumenting/pushing/) or an OTLP
+[collector](https://opentelemetry.io/docs/collector/) setup with an accessible
+URL, then you can directly switch from metric polling to metric pushing by
+passing a non `nil` argument to `autometrics.Init` for the `pushConfiguration`:
+
+``` patch
+	shutdown, err := autometrics.Init(
+		"myApp/v2/prod",
+		autometrics.DefBuckets,
+		autometrics.BuildInfo{ Version: "2.1.37", Commit: "anySHA", Branch: "", Service: "myApp" },
+-		nil,
++		&autometrics.PushConfiguration{
++			CollectorURL: "https://collector.example.com",
++			JobName: "instance_2",                         // You can leave the JobName out to let autometrics generate one
++			Period: 1 * time.Second,                       // Period is only relevant when using OpenTelemetry implementation
++			Timeout: 500 * time.Millisecond,               // Timeout is only relevant when using OpenTelementry implementation
++		},
+	)
+```
+
+> **Note**
+> If you do not want to setup an OTLP collector or a Prometheus push-gateway yourself, you
+can contact us so we can setup a managed instance of Prometheus for you. We will effectively
+give you collector URLs, that will work with both OpenTelemetry and Prometheus; and can be 
+visualized easily with our explorer as well!
+
 #### Git hook
 
 As autometrics is a Go generator that modifies the source code when run, it

docker-compose.open-telemetry-example.yaml

@@ -52,7 +52,8 @@ services:
       context: .
       dockerfile: examples/otel/Dockerfile
     environment:
-      AUTOMETRICS_SERVICE_NAME: open-telemetry-autometrics-demo
+      AUTOMETRICS_SERVICE_NAME: autometrics_otel
+      AUTOMETRICS_OTLP_URL: $AUTOMETRICS_OTLP_URL
     container_name: web-server-otel
     restart: unless-stopped
     expose:

docker-compose.prometheus-example.yaml

@@ -53,7 +53,8 @@ services:
       context: .
       dockerfile: examples/web/Dockerfile
     environment:
-      AUTOMETRICS_SERVICE_NAME: prometheus-autometrics-demo
+      AUTOMETRICS_SERVICE_NAME: autometrics_prometheus
+      AUTOMETRICS_PUSH_GATEWAY_URL: $AUTOMETRICS_PUSH_GATEWAY_URL
     container_name: web-server-prom
     restart: unless-stopped
     expose:

examples/otel/Dockerfile

@@ -10,7 +10,7 @@ COPY . ./
 
 RUN go mod download
 
-WORKDIR /app/examples/web
+WORKDIR /app/examples/otel
 
 RUN go generate cmd/main.go
 

examples/otel/cmd/main.go

@@ -6,6 +6,7 @@ import (
 	"log"
 	"math/rand"
 	"net/http"
+	"os"
 	"time"
 
 	"github.com/autometrics-dev/autometrics-go/otel/autometrics"
@@ -25,24 +26,42 @@ var (
 func main() {
 	rand.Seed(time.Now().UnixNano())
 
+	var pushConfiguration *autometrics.PushConfiguration
+	if os.Getenv("AUTOMETRICS_OTLP_URL") != "" {
+		pushConfiguration = &autometrics.PushConfiguration{
+			CollectorURL: os.Getenv("AUTOMETRICS_OTLP_URL"),
+			// NOTE: Setting the JobName is useful when you fully control the instances that will run it.
+			//   Otherwise (auto-scaling scenarii), it's better to leave this value out, and let
+			//   autometrics generate an IP-based or Ulid-based identifier for you.
+			// JobName:      "autometrics_go_otel_example",
+			Period:  1 * time.Second,
+			Timeout: 500 * time.Millisecond,
+		}
+	}
+
 	// Everything in BuildInfo is optional.
 	// You can also use any string variable whose value is
 	// injected at build time by ldflags.
-	autometrics.Init(
-		"web-server",
+	shutdown, err := autometrics.Init(
+		"web-server-go-component",
 		autometrics.DefBuckets,
 		autometrics.BuildInfo{
 			Version: Version,
 			Commit:  Commit,
 			Branch:  Branch,
 		},
+		pushConfiguration,
 	)
+	if err != nil {
+		log.Fatalf("Failed initialization of autometrics: %s", err)
+	}
+	defer shutdown(nil)
 
 	http.HandleFunc("/", errorable(indexHandler))
 	http.HandleFunc("/random-error", errorable(randomErrorHandler))
 	http.Handle("/metrics", promhttp.Handler())
 
-	log.Println("binding on http://localhost:62086")
+	log.Println("binding on http://0.0.0.0:62086")
 	log.Fatal(http.ListenAndServe(":62086", nil))
 }
 

examples/otel/go.mod

@@ -4,27 +4,39 @@ go 1.20
 
 require (
 	github.com/autometrics-dev/autometrics-go v0.0.0-20230222105517-4997cc8aa1e4
-	github.com/prometheus/client_golang v1.14.0
+	github.com/prometheus/client_golang v1.16.0
 )
 
 require (
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.1.2 // indirect
-	github.com/go-logr/logr v1.2.3 // indirect
+	github.com/cenkalti/backoff/v4 v4.2.1 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/go-logr/logr v1.2.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
-	github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
-	github.com/prometheus/client_model v0.3.0 // indirect
-	github.com/prometheus/common v0.37.0 // indirect
-	github.com/prometheus/procfs v0.8.0 // indirect
-	go.opentelemetry.io/otel v1.14.0 // indirect
-	go.opentelemetry.io/otel/exporters/prometheus v0.37.0 // indirect
-	go.opentelemetry.io/otel/metric v0.37.0 // indirect
-	go.opentelemetry.io/otel/sdk v1.14.0 // indirect
-	go.opentelemetry.io/otel/sdk/metric v0.37.0 // indirect
-	go.opentelemetry.io/otel/trace v1.14.0 // indirect
-	golang.org/x/sys v0.5.0 // indirect
-	google.golang.org/protobuf v1.28.1 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
+	github.com/oklog/ulid/v2 v2.1.0 // indirect
+	github.com/prometheus/client_model v0.4.0 // indirect
+	github.com/prometheus/common v0.44.0 // indirect
+	github.com/prometheus/procfs v0.11.1 // indirect
+	go.opentelemetry.io/otel v1.17.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlpmetric v0.40.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v0.40.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v0.40.0 // indirect
+	go.opentelemetry.io/otel/exporters/prometheus v0.40.0 // indirect
+	go.opentelemetry.io/otel/metric v1.17.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.17.0 // indirect
+	go.opentelemetry.io/otel/sdk/metric v0.40.0 // indirect
+	go.opentelemetry.io/otel/trace v1.17.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.0.0 // indirect
+	golang.org/x/net v0.10.0 // indirect
+	golang.org/x/sys v0.12.0 // indirect
+	golang.org/x/text v0.9.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
+	google.golang.org/grpc v1.57.0 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
 )
 
 replace github.com/autometrics-dev/autometrics-go => ../..