[WIP] V2 Resolved Schema #980

jsuereth · 2025-10-10T22:00:10Z

DO NOT MERGE: This is a work in progress for discussion.

This modifies the resolved telemetry schema (and template values) to look like the V2 definition schema.

With this PR you can resolve into the new schema via a new flag to check how it looks, e.g.

weaver registry resolve --v2 > test.yaml

Here's a sample of the new layout:

attributes:
  - key: android.app.state
    type:
      ...
signals:
  metrics:
  - name: aspnetcore.authentication.authenticate.duration
    instrument: histogram
    unit: s
    ...
  spans:
  - type: aws.client
    ...
  events:
  - name: app.jank
    ...
  entities:
  - type: android
    ...
refinements:
  metrics:
  - id: ...refinement id...
    ...full metric definition...
  spans:
    - id: ...refinement id...
    ...full span definition...
  events:
    - id: ...refinement id...
    ...full event definition...

lmolkova

left some comments, but looks great overall!

lmolkova · 2025-10-14T21:00:30Z

crates/weaver_forge/src/lib.rs


        assert!(diff_dir("expected_output/test", "observed_output/test").unwrap());
+
+        // TODO - Remove this.


is it ok to remove now?

I'm still using this to test/demo. I was going to remove after we lock-in on the details of the schema.

crates/weaver_forge/src/v2/metric.rs

crates/weaver_forge/src/v2/registry.rs

lmolkova · 2025-10-14T21:10:16Z

crates/weaver_forge/src/v2/registry.rs

+                instrument: metric.instrument,
+                unit: metric.unit,
+                attributes,
+                entity_associations: metric.entity_associations,


not blocking, we can follow-up: we're checking that attributes exist, but not checking it for entities, it seems inconsistent.

crates/weaver_resolved_schema/src/v2/event.rs

src/registry/resolve.rs

crates/weaver_resolved_schema/src/v2/event.rs

crates/weaver_resolved_schema/src/v2/registry.rs

thompson-tomo · 2025-10-15T12:44:42Z

crates/weaver_resolved_schema/src/v2/registry.rs

+    /// A  list of span signal definitions.
+    pub spans: Vec<Span>,
+
+    /// A  list of metric signal definitions.
+    pub metrics: Vec<Metric>,
+
+    /// A  list of event signal definitions.
+    pub events: Vec<Event>,
+
+    /// A  list of entity signal definitions.
+    pub entities: Vec<Entity>,
+
+    /// A  list of span refinements.
+    pub span_refinements: Vec<SpanRefinement>,
+
+    /// A  list of metric refinements.
+    pub metric_refinements: Vec<MetricRefinement>,
+
+    /// A  list of event refinements.
+    pub event_refinements: Vec<EventRefinement>,


I think it might be beneficial to group the definitions based on the base namespace ie db then at the top level we just have an array of namespaces. This concept and the use case for namespaces is described in more detail in https://github.com/open-telemetry/weaver/pull/867/files but I would start off with just the name + signals being the only properties of the namespace as the rest could be added via non-breaking changes.

I'm still not convinced. I see the registry itself as a namespace going forward, but this is something that needs more discussion going forward.

e.g. in instance where we want a "bundle of things" would it be easier to create a registry to describe that bundle? That registry would come with some nice properties, like versioning, isolated codegen, docs, etc.

I think I'd like to move to point where, e.g. an implementation specific registry exists in OTEL for any implementation. I think this may give us some nice properties:

Implementation-specific concerns can live near the instrumentation

Implementations can document exactly what they provide, while guaranteeing conformance to semconv

Implementations would document semconv version explicitly via their dependency to semconv repo. We could allow these to "hold back" while we do major evolutions (e.g. current RPC efforts).

There's obviously downsides too, but it's something I'd like us to consider / think through.

I like what you are thinking with regards to implementations being their own registry, but I am torn about the core sem conv registry being split. I can see it helping but also making things harder. Let's discuss implementations as standalone registry in a seperate thread.

If we look at how the namespace value would be used in the core registry, it would enable a single entry point to be provided for the namespace ie db which lists everything in that namespace rather than splitting it into a seperate entry point per signal type like we do now.

lquerel

Wow, it took me quite some time to go through all of this. Thanks for the massive refactoring of the semconv schema and everything that comes with it!

Sorry for the number of comments. A lot of them are just copy/paste issues or typos. There are probably only a handful that really concern the approach or the decisions made. Apologies in advance if some of these points or answers have already been discussed in the SIG meetings; I haven't been very present in those lately.

crates/weaver_forge/src/v2/entity.rs

lquerel · 2025-10-16T23:57:13Z

crates/weaver_forge/src/v2/entity.rs

+    /// The type of the entity.
+    pub r#type: SignalId,
+
+    /// List of attributes that belong to this event.


The comment would benefit from being updated to match the field below :-)

Apologies, I was rushing for the demo, I'll clean all these up - Didn't want this to be a full review, more of "check the direction" review.

I think the discussion on Attribute/AttributeRef is the thing to focus on in this review :)

crates/weaver_forge/src/v2/entity.rs

crates/weaver_resolved_schema/src/v2/catalog.rs

crates/weaver_resolved_schema/src/v2/mod.rs

lquerel · 2025-10-17T01:16:46Z

crates/weaver_resolved_schema/src/v2/mod.rs

+                    if let Some(a) = v2_catalog.convert_ref(attr) {
+                        span_attributes.push(span::SpanAttributeRef {
+                            base: a,
+                            requirement_level: attr.requirement_level.clone(),
+                            sampling_relevant: attr.sampling_relevant.clone(),
+                        });
+                    } else {
+                        // TODO logic error!
+                    }


Is this dance, attribute-ref -> attribute -> attribute-ref, necessary because the comparison rules for attributes differ between v1 and v2?
That brings us back to my previous question about the rationale behind this difference. In any case, once we've fully migrated to v2, we shouldn't have this kind of thing anymore.

Yes - this actually leads to many less attributes in the catalog, but the catalog now matches almost exactly what we get in Semconv attribute registry.

A lot of this dance can be fully removed when we're directly doing resolve on V2.

src/registry/resolve.rs

lquerel · 2025-10-17T01:41:00Z

crates/weaver_resolved_schema/src/v2/event.rs

+    /// the attribute is "recommended". When set to
+    /// "conditionally_required", the string provided as <condition> MUST
+    /// specify the conditions under which the attribute is required.
+    pub requirement_level: RequirementLevel,


I admit I'm not entirely clear on why requirement_level gets special treatment here. In what way is requirement_level more event-specific than note, examples, ..., in the context of a refinement?

I'm thinking of this via the "attribute registry" we have in otel.

Requirement level isn't an independent concept for any individual attribute, while examples, note is.
Requirement level only makes sense in the context of a specific signal (e.g. this metric requires this attribute). Additionally, requirement level is almost guaranteed to diverge between Metrics<->Spans/Events in some critical ways, due to cardinality constraints between the signals.

If you look at the existing semconv registries (e.g. attributes), you'll see we don't include requirement_level.

I feel pretty strongly we should NOT include requirement_level in either the attribute catalog/registry or outside of a signal context.

crates/weaver_resolved_schema/src/v2/registry.rs

…pans, no real tests.

…creating V2 schema.

crates/weaver_resolved_schema/src/lib.rs

crates/weaver_resolved_schema/src/v2/catalog.rs

crates/weaver_resolved_schema/src/v2/mod.rs

github-advanced-security

clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

codecov · 2025-10-28T13:34:18Z

Codecov Report

❌ Patch coverage is 31.43508% with 301 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.2%. Comparing base (236de8c) to head (c69a92a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/weaver_forge/src/v2/registry.rs	0.0%	223 Missing ⚠️
crates/weaver_resolved_schema/src/v2/mod.rs	59.2%	68 Missing ⚠️
crates/weaver_resolved_schema/src/v2/registry.rs	0.0%	6 Missing ⚠️
crates/weaver_resolved_schema/src/lib.rs	33.3%	2 Missing ⚠️
crates/weaver_resolved_schema/src/v2/attribute.rs	0.0%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #980     +/-   ##
=======================================
- Coverage   78.3%   75.2%   -3.1%     
=======================================
  Files         77      82      +5     
  Lines       6122    6559    +437     
=======================================
+ Hits        4795    4934    +139     
- Misses      1327    1625    +298

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

crates/weaver_resolved_schema/src/v2/catalog.rs

crates/weaver_resolved_schema/src/v2/mod.rs

…vert resolved schema to v2.

crates/weaver_resolved_schema/src/lineage.rs

crates/weaver_resolved_schema/src/v2/catalog.rs

crates/weaver_resolved_schema/src/v2/mod.rs

lmolkova · 2025-11-03T23:46:17Z

testing this out on semconv and checking against problems identified for v1 in open-telemetry/semantic-conventions#2469

Resolved schema v2 - https://gist.githubusercontent.com/lmolkova/34dc5c0b0f583ca80681af3c9334238d/raw/d5b1ce469f149c1586b277cf3fe0f5772a480b72/semconv_schema_v2_copy.yaml

works fine now !

crates/weaver_resolved_schema/src/v2/catalog.rs

+            value: None,
+            role: None,
+        });
+        assert_eq!(result.is_some(), true);


crates/weaver_resolved_schema/src/v2/catalog.rs

+            value: None,
+            role: None,
+        });
+        assert_eq!(result2.is_some(), true);


lmolkova

I tested weaver registry resolve --v2 and it works. I'm in favor of merging it and following up

lmolkova reviewed Oct 14, 2025

View reviewed changes

thompson-tomo reviewed Oct 15, 2025

View reviewed changes

lquerel reviewed Oct 17, 2025

View reviewed changes

thompson-tomo reviewed Oct 19, 2025

View reviewed changes

crates/weaver_resolved_schema/src/v2/registry.rs Show resolved Hide resolved

jsuereth added 10 commits October 20, 2025 13:56

First cut at converting from v1 resolved schema to v2. Only handles s…

59aefac

…pans, no real tests.

Add group lineage tracking so we can determine group refinement when …

54fc192

…creating V2 schema.

Fix up tests for V2 span/span_refinement conversion.

f770723

Start fixing up documentation and add metric v2 extraction.

b14dcd3

More documentation and test cleanup for spans/metrics.

c917f02

Finish implementation (not full testing) of v2 conversion for basics.

1baa3cd

Hackily hook up the v2 resolution and template for output in weaver.

a3bb434

Add spans to resolved registry.

4a9f6bf

Add the remaining missing pieces to V2 resolved schema.

edbe850

Fix all tests.

209883f

jsuereth force-pushed the wip-v2-resolved-schema branch from 67e6762 to 209883f Compare October 20, 2025 19:25

github-advanced-security bot found potential problems Oct 20, 2025

View reviewed changes

jsuereth added 2 commits October 28, 2025 09:14

Start of refactoring of 'regisry' vs. 'catalog' vs. 'refinements'

e02235c

Fix forge unwinding of registry.

a99d72e

github-advanced-security bot found potential problems Oct 28, 2025

View reviewed changes

jsuereth added 4 commits October 28, 2025 13:38

Optimise lookup up attributes in catalog when building V2.

5a01aae

Finish attribute mapping equality for v1->v2

42d4976

Fix error reporting on v2 schema creation.

1a36d26

Clippy fixes.

8889f71

github-advanced-security bot found potential problems Oct 28, 2025

View reviewed changes

Start supporting attribute groups. Add lineage tracking so we can con…

27b7a0f

…vert resolved schema to v2.

github-advanced-security bot found potential problems Oct 28, 2025

View reviewed changes

crates/weaver_resolved_schema/src/lineage.rs Fixed Show fixed Hide fixed

crates/weaver_resolved_schema/src/lineage.rs Fixed Show fixed Hide fixed

jsuereth added 2 commits October 28, 2025 16:48

Initial attribute group support in resolved registry.

75072fe

Cargo fix.

0c32132

github-advanced-security bot found potential problems Oct 28, 2025

View reviewed changes

crates/weaver_resolved_schema/src/lineage.rs Fixed Show fixed Hide fixed

crates/weaver_resolved_schema/src/lineage.rs Fixed Show fixed Hide fixed

crates/weaver_resolved_schema/src/v2/catalog.rs Fixed Show fixed Hide fixed

crates/weaver_resolved_schema/src/v2/catalog.rs Fixed Show fixed Hide fixed

jsuereth added 2 commits October 28, 2025 18:48

Add include groups to v2 resolved schema.

b7b3a48

Remove attribute group tracking from in-group feedback.

60227ac

github-advanced-security bot found potential problems Oct 29, 2025

View reviewed changes

crates/weaver_resolved_schema/src/v2/mod.rs Fixed Show fixed Hide fixed

crates/weaver_resolved_schema/src/v2/mod.rs Fixed Show fixed Hide fixed

jsuereth added 9 commits October 31, 2025 10:56

Add attribute group to forge schema.

e1a2de8

Add v2 to weaver generate and write a test.

e7617ee

cargo fmt

8b4a7bc

Fix some types and copy-paste errors.

ad75072

Fix typos.

b72b754

More spell fixes.

fb4a367

Fix typos.

dff7769

Merge remote-tracking branch 'origin/main' into wip-v2-resolved-schema

bdb91b1

Fix tests.

c56256f

jsuereth mentioned this pull request Nov 3, 2025

[project tracking] v2 Schema work to do #994

Open

15 tasks

jsuereth added 5 commits November 3, 2025 15:14

Fix format.

374c493

Clippy fixes.

d1a61fa

Fix last clippy issue.

4d22341

Fix spelling issues.

bd681a6

Cargo fmt

ce701d2

jsuereth marked this pull request as ready for review November 3, 2025 21:08

jsuereth requested a review from a team as a code owner November 3, 2025 21:08

This comment was marked as resolved.

Sign in to view

jsuereth added 2 commits November 4, 2025 08:22

Merge remote-tracking branch 'origin/main' into wip-v2-resolved-schema

fe55753

Fix logic error in comparing annotations.

c69a92a

github-advanced-security bot found potential problems Nov 4, 2025

View reviewed changes

lmolkova approved these changes Nov 4, 2025

View reviewed changes


		assert!(diff_dir("expected_output/test", "observed_output/test").unwrap());

		// TODO - Remove this.

[WIP] V2 Resolved Schema #980

Are you sure you want to change the base?

[WIP] V2 Resolved Schema #980

Uh oh!

Conversation

jsuereth commented Oct 10, 2025

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lquerel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Oct 28, 2025 •

edited

Loading

lmolkova commented Nov 3, 2025 •

edited

Loading