[Proposal] Adopt weaver for all scraping receivers

[atoulme]

Component(s)

No response

Describe the issue you're reporting

As demonstrated by #40698 it is now possible to generate metadata.yaml using Weaver.

The Weaver model can help us define our metrics in a standard manner, include semantic conventions in our metric definitions, generate documentation and downstream artifacts that can help increase the quality of our repository.

This issue aims to discuss adopting Weaver across all scraping receivers. The list of scraping receivers is derived from git grep "metrics:" | grep metadata.yaml | xargs dirname | grep receiver | xargs basename | awk '{print "* "$1}'

• activedirectorydsreceiver
• aerospikereceiver
• apachereceiver
• apachesparkreceiver
• bigipreceiver
• chronyreceiver
• couchdbreceiver
• dockerstatsreceiver
• elasticsearchreceiver
• expvarreceiver
• filestatsreceiver
• flinkmetricsreceiver
• fluentforwardreceiver
• githubreceiver
• googlecloudpubsubreceiver
• haproxyreceiver
• cpuscraper
• diskscraper
• filesystemscraper
• loadscraper
• memoryscraper
• networkscraper
• nfsscraper
• pagingscraper
• processesscraper
• processscraper
• systemscraper
• httpcheckreceiver
• iisreceiver
• k8sclusterreceiver
• kafkametricsreceiver
• kafkareceiver
• kubeletstatsreceiver
• memcachedreceiver
• mongodbatlasreceiver
• mongodbreceiver
• mysqlreceiver
• nginxreceiver
• nsxtreceiver
• ntpreceiver
• opencensusreceiver
• oracledbreceiver
• podmanreceiver
• postgresqlreceiver
• rabbitmqreceiver
• rabbitmqreceiver
• redisreceiver
• riakreceiver
• saphanareceiver
• snmpreceiver
• snowflakereceiver
• solacereceiver
• splunkenterprisereceiver
• sqlserverreceiver
• sshcheckreceiver
• statsdreceiver
• statsdreceiver
• tcpcheckreceiver
• tlscheckreceiver
• vcenterreceiver
• windowsservicereceiver

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[ChrsMark]

I support the idea of start using weaver for scraping receivers. This is related to open-telemetry/opentelemetry-collector#13297 (comment).

However, I would like to see what problem exactly weaver will solve here. Is the idea to use weaver for defining local registries for receivers in order to generate the metadata.yaml files or sth more? FYI I drafted something to check how weaver could be used in order to import metrics that are already defined in SemConv, however the solution/approach is not yet clear to me. Happy to brainstorm more on this.

[atoulme]

However, I would like to see what problem exactly weaver will solve here.

The main reason is that we have started seeing folks pushing weaver models directly to the specification (I can't find the issue now however :/) My approach is that we should derive those models in contrib, mature them, and move them to the specification eventually.

The other goal is to dogfood weaver. Using it against existing models has already helped find and fix some shortcomings.

Another goal is to use weaver for documentation and testing purposes, but I didn't get too far into this.

You can also see other experiments with weaver here: open-telemetry/opentelemetry-collector#13454 , this effort here is a bit more lightweight and just aims to generate metadata.yaml to reduce the cost, at least for now.

Please let me know if that makes sense.

[ChrsMark]

Thank's for the context @atoulme! From my perspective, one immediate goal that could be achieved through this would be to allow us linking metrics that scrapers define to semantic conventions. For example, if hostmetricsreceiver/process scraper defines process.cpu.time we would like to be able to link the "local" definition to the respsective semantic convention and apply some sort of validation on it like for example ensure that stability levels align etc.

I tried that at https://github.com/open-telemetry/opentelemetry-collector-contrib/compare/main...ChrsMark:weaver_attempt?expand=1#diff-257b58dcf0a65cf72790e116261560cb16b5136b8d4480899efc9eb6a683851cR60-R75 and it kind of worked but I couldn't apply any validation like the one I mentioned. @jsuereth shared in weaver's slack channel that there are plans about improving the support for this.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.