Skip to content

Commit 5d5e141

Browse files
ryanroldsryanrolds
authored
Merge pull request #5 from solo-io/rolds/test-metrics
Metrics for test cases
2 parents 80a05ac + 24d6bfc commit 5d5e141

File tree

5 files changed

+254
-121
lines changed

5 files changed

+254
-121
lines changed

‎.github/workflows/tests.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ on:
66
- main
77
pull_request:
88
branches:
9-
- main
9+
- '*'
1010

1111
jobs:
1212
build:

‎README.md

+143-85
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,146 @@
22

33
[![Tests](https://github.com/mdelapenya/junit2otlp/actions/workflows/tests.yml/badge.svg)](https://github.com/mdelapenya/junit2otlp/actions/workflows/tests.yml)
44

5-
This simple CLI, written in Go, is sending jUnit metrics to a back-end using [Open Telemetry](https://opentelemetry.io).
5+
This simple CLI, written in Go, is sending jUnit metrics to a back-end using [OpenTelemetry](https://opentelemetry.io).
66

77
> Inspired by https://github.com/axw/test2otlp, which sends traces and spans for `go test` JSON events as they occur.
88
99
## Background
10-
As jUnit represents a de-facto standard for test results in every programming language, this tool consumes the XML files produced by the test runner (or a tool converting to xUnit format), sending metrics to one or more open-source or commercial back-ends with Open Telemetry.
10+
11+
As jUnit represents a de-facto standard for test results in every programming language, this tool consumes the XML files produced by the test runner (or a tool converting to xUnit format), sending metrics to one or more open-source or commercial back-ends with OpenTelemetry.
12+
13+
## Configuration
14+
15+
This tool is able to override the following attributes:
16+
17+
| Attribute | Flag | Default value | Description |
18+
| --------- | ---- | ------------- | ----------- |
19+
| Repository Path | --repository-path | `.` | Path to the SCM repository to be read. |
20+
| Service Name | --service-name | `junit2otlp` | Overrides OpenTelemetry's service name. If the `OTEL_SERVICE_NAME` environment variable is set, it will take precedence over any other value. |
21+
| Service Version | --service-version | Empty | Overrides OpenTelemetry's service version. If the `OTEL_SERVICE_VERSION` environment variable is set, it will take precedence over any other value. |
22+
| Trace Name | --trace-name | `junit2otlp` | Overrides OpenTelemetry's trace name. |
23+
| Properties Allowed | --properties-allowed | All | Comma separated list of properties to be allowed in the jUnit report. |
24+
| Skip Sending Traces | --skip-traces | `false` | Skip sending traces to the OpenTelemetry collector. |
25+
| Skip Sending Metrics | --skip-metrics | `false` | Skip sending metrics to the OpenTelemetry collector. |
26+
27+
For using this tool in a distributed tracing scenario, where there is a parent trace in which the test reports traces should be attached, it's important to set the `TRACEPARENT` environment variable, so that the traces and spans generated by this tool are located under the right parent trace. Please read more on this [here](https://github.com/open-telemetry/opentelemetry-specification/issues/740).
28+
29+
For further reference on environment variables in the OpenTelemetry SDK, please read the [official specification](https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/)
30+
31+
## Traces
32+
33+
[Traces](https://opentelemetry.io/docs/concepts/signals/traces/) are sent to the OpenTelemetry collector, representing the test execution. Each run of the tool will create a root trace that contains spans for each suite. Each suite will contain spans for each test case.
34+
35+
## Metrics
36+
37+
[Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/) are sent to the OpenTelemetry collector, representing the test execution. Each run of the tool will create a set of metrics for the test execution, including the number of failed, errored, skipped, and passed tests, the total number of tests, and the duration of the test execution.
38+
39+
| Metric | Description |
40+
| --------- | ----------- |
41+
| `tests.suite.failed` | Number of failed tests in the test execution |
42+
| `tests.suite.error` | Number of errored tests in the test execution |
43+
| `tests.suite.skipped` | Number of skipped tests in the test execution |
44+
| `tests.suite.passed` | Number of passed tests in the test execution |
45+
| `tests.suite.total` | Total number of tests in the test execution |
46+
| `tests.suite.duration` | Duration of the test execution |
47+
| `tests.suite.duration.histogram` | Histogram of the test execution duration |
48+
| `tests.case.failed` | The test failed |
49+
| `tests.case.error` | The test errored |
50+
| `tests.case.skipped` | The test errored |
51+
| `tests.case.passed` | The test passed |
52+
| `tests.case.duration` | Duration of the test execution |
53+
| `tests.case.duration.histogram` | Histogram of the test execution duration |
54+
55+
## OpenTelemetry Attributes
56+
57+
[Attributes](https://opentelemetry.io/docs/specs/otel/common/#attribute) are added to the traces, spans and metrics sent to the OpenTelemetry collector. They are used to provide context to the test execution, and to correlate the test execution with the authors of the test suite and test cases.
58+
59+
The attributes are divided into five categories:
60+
61+
- Runtime attributes
62+
- Ownership attributes
63+
- Report properties
64+
- Test suite attributes
65+
- Test case attributes
66+
67+
### Runtime attributes
68+
69+
Runtime attributes are added to the root trace, spans, and metrics sent by the tool.
70+
71+
| Attribute | Description |
72+
| --------- | ----------- |
73+
| `host.arch` | Architecture of the host where the test execution is processed |
74+
| `os.name` | Name of the OS where the test execution is processed |
75+
| `service.name` | Name of the service where the test execution is processed |
76+
| `service.version` | Version of the service where the test execution is processed |
77+
78+
### Ownership attributes
79+
80+
These attributes are added to the traces, spans and metrics, identifying the owner (or owners) of the test suite, trying to correlate a test failure with an author or authors. To identify the owner, the tool will inspect the SCM repository for the project.
81+
82+
#### SCM attributes
83+
84+
Because the XML test report is evaluated for a project **in a SCM repository**, the tool will add the following attributes to each trace and span:
85+
86+
| Attribute | Description |
87+
| --------- | ----------- |
88+
| `scm.authors` | Array of unique Email addresses for the authors of the commits |
89+
| `scm.baseRef` | Name of the target branch (Only for change requests) |
90+
| `scm.branch` | Name of the branch where the test execution is processed |
91+
| `scm.committers` | Array of unique Email addresses for the committers of the commits |
92+
| `scm.provider` | Optional. If present, will include the name of the SCM provider, such as Github, Gitlab, Bitbucket, etc. |
93+
| `scm.repository` | Array of unique URLs representing the repository (i.e. https://github.com/mdelapenya/junit2otlp) |
94+
| `scm.type` | Type of the SCM (i.e. git, svn, mercurial) At this moment the tool only supports Git repositories. |
95+
96+
#### Change request attributes
97+
98+
The tool will add the following attributes to each trace, span, and metric if and only if the XML test report is evaluated in the context of a change requests **for a Git repository**:
99+
100+
| Attribute | Description |
101+
| --------- | ----------- |
102+
| `scm.git.additions` | Number of added lines in the changeset |
103+
| `scm.git.deletions` | Number of deleted lines in the changeset |
104+
| `scm.git.clone.depth` | Depth of the git clone |
105+
| `scm.git.clone.shallow` | Whethere the git clone was shallow or not |
106+
| `scm.git.files.modified` | Number of modified files in the changeset |
107+
108+
A changeset is calculated based on the HEAD commit and the first ancestor between HEAD and the branch where the changeset is submitted against.
109+
110+
### Report properties
111+
112+
The jUnit XML report can contain properties at different levels. The tool will add the properties to the testsuite and testcase spans automatically. If the `--properties-allowed` flag is set, only the properties listed in the flag will be added to the spans.
113+
114+
### Test suite attributes
115+
116+
For each test suite in the test execution, the tool will add the following attributes to the span document representing the test suite:
117+
118+
| Attribute | Spans | Metrics | Description |
119+
| --------- | ----- | ------- | ----------- |
120+
| `code.namespace` | x | x | Class/module of the test suite |
121+
| `tests.suite.suitename` | x | x | Name of the test suite |
122+
| `tests.suite.duration` | x | | Duration of the test suite |
123+
| `tests.suite.systemerr` | x | | Log produced by Systemerr |
124+
| `tests.suite.systemout` | x | | Log produced by Systemout |
125+
126+
### Test case attributes
127+
128+
For each test case in the test execution, the tool will add the following attributes to the span document representing the test case:
129+
130+
| Attribute | Spans | Metrics | Description |
131+
| --------- | ----- | ------- | ----------- |
132+
| `code.namespace` | x | x | Class/module of the test suite |
133+
| `code.function` | x | x | Function or method of the test case |
134+
| `tests.suite.suitename` | x | x | Name of the test suite |
135+
| `tests.case.classname` | x | x | Classname or file for the test case |
136+
| `tests.case.duration` | x | | Duration of the test case |
137+
| `tests.case.error` | x | | Error message of the test case |
138+
| `tests.case.message` | x | | Message of the test case |
139+
| `tests.case.status` | x | | Status of the test case |
140+
| `tests.case.systemerr` | x | | Log produced by Systemerr |
141+
| `tests.case.systemout` | x | | Log produced by Systemout |
11142

12143
## Supported CI runners
144+
13145
This tool will work in the context of a CI runner, such as a Github action, a Jenkins job, a Gitlab runner, or even a local execution. This is important because it will use the context of the CI execution to infer the attributes to be added to the OpenTelemetry traces and spans.
14146

15147
In particular the order of evaluation to detect the right execution context is the following:
@@ -19,6 +151,7 @@ In particular the order of evaluation to detect the right execution context is t
19151
```
20152

21153
### Local execution
154+
22155
It reads the environment variables that are avaible in the context of a local execution, representing the fallback if no context is discovered:
23156

24157
```golang
@@ -73,6 +206,7 @@ func FromGithub() *ScmContext {
73206
```
74207

75208
### Jenkins multibranch pipelines
209+
76210
It reads the environment variables that are avaible in the context of a Jenkins multibranch pipeline execution:
77211

78212
```golang
@@ -109,6 +243,7 @@ func FromJenkins() *ScmContext {
109243
```
110244

111245
### Gitlab Runners
246+
112247
It reads the environment variables that are avaible in the context of a Gitlab runner execution:
113248

114249
```golang
@@ -136,90 +271,8 @@ func FromGitlab() *ScmContext {
136271
}
137272
```
138273

139-
## OpenTelemetry configuration
140-
This tool is able to override the following attributes:
141-
142-
| Attribute | Flag | Default value | Description |
143-
| --------- | ---- | ------------- | ----------- |
144-
| Repository Path | --repository-path | `.` | Path to the SCM repository to be read. |
145-
| Service Name | --service-name | `junit2otlp` | Overrides OpenTelemetry's service name. If the `OTEL_SERVICE_NAME` environment variable is set, it will take precedence over any other value. |
146-
| Service Version | --service-version | Empty | Overrides OpenTelemetry's service version. If the `OTEL_SERVICE_VERSION` environment variable is set, it will take precedence over any other value. |
147-
| Trace Name | --trace-name | `junit2otlp` | Overrides OpenTelemetry's trace name. |
148-
| Properties Allowed | --properties-allowed | All | Comma separated list of properties to be allowed in the jUnit report. |
149-
| Skip Sending Traces | --skip-traces | `false` | Skip sending traces to the OpenTelemetry collector. |
150-
| Skip Sending Metrics | --skip-metrics | `false` | Skip sending metrics to the OpenTelemetry collector. |
151-
152-
For using this tool in a distributed tracing scenario, where there is a parent trace in which the test reports traces should be attached, it's important to set the `TRACEPARENT` environment variable, so that the traces and spans generated by this tool are located under the right parent trace. Please read more on this [here](https://github.com/open-telemetry/opentelemetry-specification/issues/740).
153-
154-
For further reference on environment variables in the OpenTelemetry SDK, please read the [official specification](https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/)
155-
156-
## OpenTelemetry Attributes
157-
This tool is going to parse the XML report produced by jUnit, or any other tool converting to that format, adding different attributes, separated by different categories:
158-
159-
- Test metrics attributes
160-
- Ownership attributes
161-
162-
### Metrics and Traces
163-
The following attributes are added as metrics and/or traces.
164-
165-
#### Test execution attributes
166-
For each test execution, represented by a test report file, the tool will add the following attributes to the metric document, including them in the trace representing the test execution.
167-
168-
| Attribute | Description |
169-
| --------- | ----------- |
170-
| `tests.suite.failed` | Number of failed tests in the test execution |
171-
| `tests.suite.error` | Number of errored tests in the test execution |
172-
| `tests.suite.passed` | Number of passed tests in the test execution |
173-
| `tests.suite.skipped` | Number of skipped tests in the test execution |
174-
| `tests.suite.duration` | Duration of the test execution |
175-
| `tests.suite.suitename` | Name of the test execution |
176-
| `tests.suite.systemerr` | Log produced by Systemerr |
177-
| `tests.suite.systemout` | Log produced by Systemout |
178-
| `tests.suite.total` | Total number of tests in the test execution |
179-
180-
#### Test case attributes
181-
For each test case in the test execution, the tool will add the following attributes to the span document representing the test case:
182-
183-
| Attribute | Description |
184-
| --------- | ----------- |
185-
| `tests.case.classname` | Classname or file for the test case |
186-
| `tests.case.duration` | Duration of the test case |
187-
| `tests.case.error` | Error message of the test case |
188-
| `tests.case.message` | Message of the test case |
189-
| `tests.case.status` | Status of the test case |
190-
| `tests.case.systemerr` | Log produced by Systemerr |
191-
| `tests.case.systemout` | Log produced by Systemout |
192-
193-
### Ownership attributes
194-
These attributes are added to the traces and spans sent by the tool, identifying the owner (or owners) of the test suite, trying to correlate a test failure with an author or authors. To identify the owner, the tool will inspect the SCM repository for the project.
195-
196-
#### SCM attributes
197-
Because the XML test report is evaluated for a project **in a SCM repository**, the tool will add the following attributes to each trace and span:
198-
199-
| Attribute | Description |
200-
| --------- | ----------- |
201-
| `scm.authors` | Array of unique Email addresses for the authors of the commits |
202-
| `scm.baseRef` | Name of the target branch (Only for change requests) |
203-
| `scm.branch` | Name of the branch where the test execution is processed |
204-
| `scm.committers` | Array of unique Email addresses for the committers of the commits |
205-
| `scm.provider` | Optional. If present, will include the name of the SCM provider, such as Github, Gitlab, Bitbucket, etc. |
206-
| `scm.repository` | Array of unique URLs representing the repository (i.e. https://github.com/mdelapenya/junit2otlp) |
207-
| `scm.type` | Type of the SCM (i.e. git, svn, mercurial) At this moment the tool only supports Git repositories. |
208-
209-
#### Change request attributes
210-
The tool will add the following attributes to each trace and span if and only if the XML test report is evaluated in the context of a change requests **for a Git repository**:
211-
212-
| Attribute | Description |
213-
| --------- | ----------- |
214-
| `scm.git.additions` | Number of added lines in the changeset |
215-
| `scm.git.deletions` | Number of deleted lines in the changeset |
216-
| `scm.git.clone.depth` | Depth of the git clone |
217-
| `scm.git.clone.shallow` | Whethere the git clone was shallow or not |
218-
| `scm.git.files.modified` | Number of modified files in the changeset |
219-
220-
A changeset is calculated based on the HEAD commit and the first ancestor between HEAD and the branch where the changeset is submitted against.
221-
222274
## Docker image
275+
223276
It's possible to run the binary as a Docker image. To build and use the image
224277

225278
1. First build the Docker image using this Make goal:
@@ -242,6 +295,7 @@ cat TEST-sample3.xml | docker run --rm -i --network elastic_junit2otlp --volume
242295
- We are passing command line flags to the container, setting the service name (_DOCKERFOO_) and the trace name (_TRACEBAR_).
243296

244297
## Demos
298+
245299
To demonstrate how traces and metrics are sent to different back-ends, we are provising the following demos:
246300

247301
- Elastic
@@ -250,6 +304,7 @@ To demonstrate how traces and metrics are sent to different back-ends, we are pr
250304
- Zipkin
251305

252306
### Elastic
307+
253308
It will use the Elastic Stack as back-end, sending the traces, spans and metrics through the APM Server, storing them in Elasticsearch and finally using Kibana as visualisation layer.
254309

255310
```shell
@@ -262,6 +317,7 @@ open http://localhost:5601/app/apm/services?rangeFrom=now-15m&rangeTo=now&compar
262317
```
263318

264319
### Jaeger
320+
265321
It will use Jaeger as back-end, sending the traces, spans and metrics through the OpenTelemetry collector, storing them in memory.
266322

267323
```shell
@@ -274,6 +330,7 @@ open http://localhost:16686
274330
```
275331

276332
### Prometheus
333+
277334
It will use Prometheus as back-end, sending the traces, spans and metrics through the OpenTelemetry collector, storing them in memory.
278335

279336
```shell
@@ -286,6 +343,7 @@ open http://localhost:9090
286343
```
287344

288345
### Zipkin
346+
289347
It will use Prometheus as back-end, sending the traces, spans and metrics through the OpenTelemetry collector, storing them in memory.
290348

291349
```shell

0 commit comments

Comments
 (0)