Update naming conventions for multi-registry manifests

[jsuereth]

As part of #1160 - We discussed certain differences / naming conventions from what Weaver has today vs. what we want in the OTEP. A few points to nail down and change:

• Migrating from registry_manifest.yaml to just manifest.yaml. We should (in a non-breaking way) move to the simpler named file.
• Consistency in referencing registries. We have _id, _uri and _name today, we need to sort out what these mean.
• Dependency Resolution needs to understand if we have a "diamond dependency". This means we need to see if we're depending on "the same" registry but two different versions.

A set of principles:

• We need a name for a registry. This should be unique across all registries, and can be used to denote if we have a dependency chain with two versions of the "same" registry.
• We need a uri for a registry. This should denote the Manifest file or the directory where the manifest file can be found.
• We need a uri for where to find resolved schema. This should denote where we load the resolved schema and will be a VirtualDirectoryRef. (Note: VirtualDirectoryRef can be a file in a virtual directory, e.g. a file in a git repo or a file inside a zip archive).

Strawman Proposal:

• Allow registry_manfiest.yaml or manfiest.yaml to be the name of a registry manifest.
  • Prefer manifest.yaml if it exists.
  • Issue a warning if we see registry_manifest.yaml is used.
• We consistently use "Weaver URI" in documentation and code to represent VirtualDirectoryRef strings.
• Change RegistryManifest as follows:
  • resolved_schema_url becomes resolved_schema_uri.
• name is removed.
• repository_url is changed to be schema_url - I.e. this is the schema_url this registry will fill in telemetry.
• name of a registry will be inferred from schema_url - We drop the "version" component of the URL, and we get a unique name we can use to identify version conflicts in dependencies.

[lmolkova]

Most of it seems great, but I'm trying to understand how it'd play out in different cases and not sure we can put that much expectations on the schema_url and require it to be in a certain format.

let's say I have a local semconv, not published

# manifest.yaml
schema_url: ./model
resolve_schema_uri: ./resolved-schema.yaml

It's a valid manifest and it could be ok to depend on it.

---

Let's say:

• in the RegistryManifest:
  • name and version are required
  • let's drop repository_url or emphasize it's completely optional
• in the Dependency:
  • remove name
    • we could allow friendly dependency name, but unless it matches registry name in manifest, it'd be misleading
  • use provided registry_uri to get registry name and version from the manifest file
• we'd probably add name and version to provenance later

This way:

• we don't depend on schema_url formatting.
• we can use unpublished schemas as dependencies (I don't feel strong about it being important in production cases, but should be useful for development)

skip to #1197 (comment)

[lmolkova]

skip to #1197 (comment)

Definition manifest:

• name - required
• version - required
• dependencies - optional
  • registry_uri - required, can be published schema_url or local manifest

Resolved manifest: (get it by running weaver registry pack(?) -r {registry_uri})

• name - required
• version - required
• resolved_schema_uri - local path by default, will need config/rewrite to know where it WILL be published

Provenance:

• registry_name: required
• registry_version: required
• // there will be other properties, so adding registry_ to disambiguate
• // alternative: schema_url but it requires all dependencies to be published and creates coupling between what's published and where/how
• // another alternative: source_registry: - name -version

Resolved /materialized schema

• provenance (for this registry)
• attributes:
  • key
  • ...
  • provenance (for each attribute and signal definition/refinement). Can potentially skip if originating from this registry to save bandwidth

e.g.

file_format: resolved/2.0.0
provenance:
  registry_name: acme/schemas
  registry_version: 1.1.0
# or
# source_registry:
#  name: acme/schemas
#  version: 1.1.0
attribute_catalog:
...
- key: server.address
  ...
  provenance:
     - registry_name: open-telemetry/semantic-conventions
     - registry_version: 1.40.0
  # or
  # source_registry:
  #   name: open-telemetry/semantic-conventions
  #   version: 1.40.0
- key: my.operation.name
  type: string
  stability: development
  brief: My service operation name as defined in this Open API spec
  examples: ["get_object", "create_collection"]
  # no provenance, default to top-level one
...
registry:
  attributes:
  ...
  - 888   # this is the index of `server.address` in the attribute_catalog
  - 1042  # this is the index of `my.operation.name` in attribute_catalog
  ...
  metrics:
  - name: http.client.request.duration
    ...
    provenance:
     - registry_name: open-telemetry/semantic-conventions
     - registry_version: 1.40.0
  - name: my.client.operation.duration
    instrument: histogram
    unit: s
    attributes:
      - base: 1042  # this is the index of `my.operation.name` in attributes list
        requirement_level: required
      - base: 888  # this is the index of `server.address` in the attributes list
        requirement_level: recommended
      ...
    # no provenance, default to top-level one

[lmolkova]

or even

# definition manifest
name: acme/schemas
version: 1.1.0
dependencies: 
  - registry_uri: http://otel.io/schemas/semconv/1.40.0 # or https://github.com/open-telemetry/semantic-conventions/archive/refs/tags/v1.40.0.zip[model] or even ./model (with limitations)

# resolved-schema.yaml
file_format: resolved/2.0.0
manifest: 
  name: acme/schemas
  version: 1.1.0
  dependencies: 
    # if it's a local path, nobody else can depend on this schema - we should fail if we discover local path we can't access
    -  registry_uri: http://otel.io/schemas/semconv/1.40.0 # or https://github.com/open-telemetry/semantic-conventions/archive/refs/tags/v1.40.0.zip[model]
      # name: open-telemetry/semantic-conventions # no need, can extract from registry_uri
      # version: 1.40.0                           # no need, can extract from registry_uri
    ...
    # all (including transient) dependencies that have active references appear in this list
    # name is unique - one version per name, latest wins
attribute_catalog:
- key: server.address
  ...
  provenance:
     - dependency: 0
...

[UPDATE]
We can't make transitive dependencies work with local registry_uri - if someone takes dependency on acme/schemas, resolution process needs to know where to get full schema for all dependencies.

E.g. attribute foo.bar is present in some-registry v1.0.0-dev and is removed in v1.1.0-dev, let's say some-registry is a transitive dependency. When we resolve foo.bar, we need to look for it in the 1.1.0-dev and fail if it's not present, so we need to access full resolved schema of all dependencies

[jsuereth]

When I suggested schema url, I meant the one specified in the otel specification. I agree it cannot be a local thing, schema url needs to be both and identifying name and a place we look for the contents of the identifier

I see a world where weaver is given a local file for --registry and this will still resolve a formal schema url from the manifest for which we generate the identity of the dependency.

I need to read the rest of your writing but wanted to confirm I hadn't even thought about allowing a schema url that is "local".

[lmolkova]

weaver sig call discussion

# definition manifest
schema_url: http://acme/schemas/1.1.0 # this does not have to be actual URL, there could be a separate config that tells what's the actual location, like custom maven repository 
dependencies: 
  - schema_url: http://otel.io/schemas/semconv/1.40.0
    # can provide additional properties here: alias, actual URI to get this from 

# resolved-schema.yaml
file_format: resolved/2.0.0
schema_url: http://acme/schemas/1.1.0
dependencies: 
  - schema_url: http://otel.io/schemas/semconv/1.40.0
attribute_catalog:
- key: server.address
  ...
  provenance:
     - dependency: 0
...