Contribution of the resource providers to opentelemetry-java-contrib repo

[SylvainJuge]

Hi !

I just opened open-telemetry/opentelemetry-java-contrib#1074 in the java contrib repo after discussing the topic in the Java SIG meeting.

Ideally we'd like to have the most common cloud resource provider implementations available, would you be interested in contributing your implementation currently in detectors/resources to the open-telemetry/opentelemetry-java-contrib repo ?

If yes, would you also be interested in being maintainers of this component within otel ? That does not sound like a very strong commitment and you'll benefit from frequent opentelemetry release schedule and automatic version upgrades. I can definitely be a second co-maintainer of this component within otel if needed.

I saw that this project currently relies on the now obsolete io.opentelemetry:opentelemetry-sdk-extension-resources:1.19.0 dependency, which has now been moved to opentelemetry-java-contrib repository, as a consequence replacing this will already create a dependency to the contrib repo.
What I mean here is that ideally a single implementation in the contrib repo could easily be reused here without adding an extra repository as a dependency (just an extra artifact).

[SylvainJuge]

For now I see two potential aspects that could be problematic for re-use in OTel-based agents or SDKs:

• resource attributes fetched on demand and one by one, which means more than one HTTP request to the metadata endpoint which could make startup slower.
• currently targets Java 11, whereas upstream opentelemetry is java 8 compatible.

[dashpole]

Hey @SylvainJuge, thanks for reaching out. Overall, we would be happy to contribute the resource detector here to the java contrib repo.

One of our original reasons for having it in this repository was to be able to run presubmit integration tests on GCP infrastructure to ensure the resource detectors work as expected. For other languages, like golang, we ended up splitting the detector into a core detection library (https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/tree/main/detectors/gcp), and a contrib entrypoint (https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/detectors/gcp/detector.go). We can run integration tests on the core library to ensure the detection methods work properly, but the actual detector would live in the upstream contrib repository. Would something like that work for you?

From our side, @psx95 and @jsuereth would be code owners for the contrib detector, and @psx95 will work on the new detector in contrib, including working through the issues you mention.

[SylvainJuge]

I see, having integration test on your side definitely makes sense to ensure everything is working as expected in the target environment and to keep a short feedback loop.

If I understand correctly the plan you suggest involves:

• split the current resource detector in a "library" and a "resource detector"
• keep the library in this repository so you can easily maintain it and keep your integration tests
• contribute the resource detector to the opentelemetry contrib repo, this will just delegate to the library and means having a dependency on it.
• your current resource detector could be replaced by the contrib one (which delegates to the library).

From the perspective of a consumer of this GCP resource detector:

• the resource detector brings a dependency to the library
• the library should be minimal and should not bring extra dependencies* and be java 8 compatible.

* : for the AWS equivalent it used to use the complete AWS SDK and is now rewritten with a few HTTP queries.

Another alternative could be to keep everything in one "library+resource detector" in the contrib repo:

• also adds a dependency to the contrib repo
• you can keep your current integration tests as they are
• tests would still be required on the contrib repo, probably by mocking a few HTTP endpoints like existing resource detectors in the contrib repo.
• it will be released at the same frequency as other opentelemetry components

But that second option would likely introduce more friction than necessary on your side if you need to change anything the library, thus the first is probably better here.

Ideally, without going too deep into the implementation details, the "library" could be just about parsing the metadata, without any dependencies. That would help for testing without HTTP communication and using it with another HTTP client (see open-telemetry/opentelemetry-java-contrib#1061 for context).

[dashpole]

As a first step, @psx95 can work on a PR to extract the library into its own package here. Then, we can take a look at the dependencies, and see if it works, or if we need to explore other options.

The list of dependencies seems reasonable (at least no GCP SDKs, or anything), so hopefully it shouldn't be too hard:

opentelemetry-operations-java/detectors/resources/build.gradle

Lines 18 to 29 in 25e1b88

	dependencies {
	implementation(libraries.opentelemetry_api)
	implementation(libraries.opentelemetry_sdk)
	implementation(libraries.opentelemetry_sdk_autoconf)
	implementation(libraries.opentelemetry_semconv)
	implementation platform(libraries.opentelemetry_bom)
	testImplementation(testLibraries.assertj)
	testImplementation(testLibraries.junit)
	testImplementation(testLibraries.wiremock)
	testImplementation(testLibraries.mockito)
	testImplementation(testLibraries.opentelemetry_sdk_testing)
	}

[psx95]

I have a draft PR - #276 which refactors the platform detection logic from the resource providers and allows resource provider to rely on the library methods to retrieve the values for the attributes. (This is in-line with what our Go resource detector does)

However, which attributes to add to a particular resource after it has been detected would still be specified in the resource provider (proposed to go in contrib repo). Just wanted to call out the following effects of this approach -

1. The support library does not need to take dependency on OTel APIs, which means the upstream library can be kept up-to-date with latest APIs without being hindered by the need to update support library dependencies.
2. Our integration tests passing relies on correct detection logic (support lib) and correct attributes being present (upstream contrib repo).

Let me know if this approach works, and I can get toward finalizing the draft PR (notably, the unit tests are currently missing).

[psx95]

@SylvainJuge would you mind taking a look at #276 - it splits our resource detection into 2 modules -

1. support library: detectors/resources-support
2. upstream detector: detectors/resources

Let us know if this split works for the upstream contribution.