Add trait to declare types for metadata

This adds a new trait to declare types for metadata keys. Values for
these keys will be validated against the shape targeted by the trait.

While this enables modeling of the suppressions and validators keys,
actually adding those to the model will be done later.

Author: JordonPhillips

.changes/next-release/feature-5c018acaef9147e7499e70c6d8919d6cf0b496c5.json

@@ -0,0 +1,5 @@
+{
+  "type": "feature",
+  "description": "Added a new `metadata` trait that allows model authors to declare types for metadata keys that will be automatically validated when building models.",
+  "pull_requests": []
+}

designs/typed-metadata.md

@@ -0,0 +1,172 @@
+# Typed Metadata
+
+Metadata is a schema-less extensibility mechanism used to associate metadata to
+an entire model. For example, metadata is used to define validators and
+model-wide suppressions.
+
+This document describes a way to define typing information for metadata that is
+automatically validated by Smithy.
+
+## Motivation
+
+The schema-less nature of metadata has allowed it to be used for any purpose
+without much hassle. However, that has come at the cost of increased validation
+complexity. Any tool, including Smithy itself, that uses metadata in a
+structured way has to perform validation itself. By allowing opt-in validation,
+we can centralize and deduplicate that effort.
+
+## Proposal
+
+Model authors may globally declare the type of a metadata key by targeting a
+shape with the `@metadata` trait.
+
+```smithy
+/// Defines a type for a metadata key.
+///
+/// If a matching key is defined in the model, its value will be validated
+/// according to the targeted shape.
+///
+/// The type for any metadata key MUST only be defined once.
+@trait(selector: "dataType :not([trait|input]) :not([trait|output])")
+structure metadata {
+    /// The metadata key to validate. Each key MUST only be defined once.
+    @required
+    @length(min: 1)
+    key: String
+}
+```
+
+For example, the
+[suppressions metadata](https://smithy.io/2.0/spec/model-validation.html#suppressions-metadata)
+could be defined as:
+
+```smithy
+$version: "2.0"
+
+namespace smithy.api
+
+@metadata(key: "suppressions")
+list MetadataSuppressions {
+    member: MetadataSuppression
+}
+
+structure MetadataSuppression {
+    @required
+    id: String
+
+    @required
+    @pattern("^(\*|[_a-zA-Z]\w*(\.[_a-zA-Z]\w*)*)$")
+    namespace: String
+
+    reason: String
+}
+```
+
+### Validation
+
+A metadata key MUST NOT be defined more than once with the `@metadata` trait. If
+a metadata key is defined more than once with the `@metadata` trait, an
+`ERROR`-level validation event will be emitted.
+
+Validation of metadata values will behave no differently than validation
+anywhere else, with severity levels being determined by each individual
+validator.
+
+Shapes targeted by the metadata shape and shapes transitively referenced by
+those shapes will not trigger the unreferenced shapes validator.
+
+Adding or removing this trait is considered backwards-compatible because
+metadata does not inherently change any interface. It may cause a build to break
+if the new validation rejects existing metadata, but that is intended behavior.
+
+## Alternatives
+
+### Inline definitions
+
+The metadata trait globally defines the shape of a metadata key, but we could
+allow local, inline definitions additionally or instead. This can be achieved by
+using a special `$type` key in metadata's node value.
+
+```smithy
+$version: "2.0"
+
+metadata foo = {
+    "$type": "com.example#Foo"
+    "bar": "baz"
+}
+
+namespace com.example
+
+structure Foo {
+    bar: String
+}
+```
+
+The problem with this strategy is that it can only be applied to structure and
+union shapes, because other shape types have nowhere to add the `$type` key or
+it would potentially shadow a valid value key.
+
+Other shapes could instead be defined via a `__type__` metadata key:
+
+```smithy
+$version: "2.0"
+
+metadata "__type__": [
+    ["foo", "com.example#Foo"]
+]
+
+metadata foo = "bar"
+
+namespace com.example
+
+string Foo
+
+@metadata(key: "__type__")
+list MetdataTypeDeclarations {
+    member: MetadataTypeDeclaration
+}
+
+@length(min: 2, max: 2)
+list MetadataTypeDeclaration {
+    member: String
+}
+```
+
+Unfortunately, this is inherently a global declaration that is not inline with
+the value, defeating the intent of an inline definition. The metadata trait is
+better suited to this purpose, as it is clearer, easier to validate, more easily
+discoverable, and more easily trackable.
+
+Smithy 2.1 could introduce new syntax to declare inline metadata types that isn't
+tied to the value, but the AST could not support it without either exposing the
+same problem or making a breaking change.
+
+## FAQ
+
+### Does this interact with metadata merging semantics?
+
+When a given metadata key is defined more than once, it usually results in a
+conflict that fails the build. If both definitions are arrays, however, they are
+instead merged. If a metadata key is defined as a list, the merged array will be
+validated.
+
+This trait does not change merging semantics.
+
+### Will the existing defined metadata keys get metadata-trait-based definitions?
+
+Smithy currently has definitions for three metadata keys: `severityOverrides`,
+`suppressions`, and `validators`. All three could be defined with the metadata
+trait.
+
+However, these three keys are currently parsed and validated early on when
+loading a model, before validation generally is run. They will need to be
+special-cased to either run early or to be skipped in favor of existing
+validation.
+
+These keys will initially be reserved so they can't be defined. As the keys get
+full definitions, their reserve list entries will be removed.
+
+### What happens if a defined metadata key is not used?
+
+Nothing. Metadata keys are inherently optional. A `required` member may be added
+later to enforce presence if there is a need.

docs/source-2.0/spec/model-validation.rst

@@ -867,3 +867,86 @@ Validator definition
       - ``string``
       - The :ref:`severity <severity-definition>` to use when an
         incompatible shape is found. Defaults to ``ERROR`` if not set.
+
+
+.. smithy-trait:: smithy.api#metadata
+.. _metadata-trait:
+
+------------------
+``metadata`` trait
+------------------
+
+:ref:`Metadata <metadata>` is schema-less by default. Any tool that consumes
+metadata, including Smithy itself, must validate it independently. The
+``metadata`` trait lets model authors define a type for a metadata key.
+When the type for a metadata key is defined this way, Smithy will automatically
+validate any value for that metadata key against its defined type.
+
+Summary
+    Defines a type for a metadata key by targeting a shape. When a metadata
+    entry with a matching key is defined in the model, its value is
+    validated against the targeted shape.
+Trait selector
+    ``dataType :not([trait|input]) :not([trait|output])``
+
+    *A :ref:`simple shape <simple-types>` or
+    :ref:`aggregate shape <aggregate-types>` that is not marked with the
+    :ref:`input trait <input-trait>` or :ref:`output trait <output-trait>`.*
+Value type
+    ``structure``
+
+The ``metadata`` trait is a structure that contains the following members:
+
+.. list-table::
+    :header-rows: 1
+    :widths: 10 10 80
+
+    * - Property
+      - Type
+      - Description
+    * - key
+      - ``string``
+      - **Required**. The metadata key whose type is being defined. The
+        type of any metadata key may only be defined once across the entire
+        model. This value must be non-empty.
+
+.. rubric:: Example
+
+Consider a service that configures an ``auditLevel`` metadata entry. Defining
+a type for the key gives Smithy enough information to validate each usage:
+
+.. code-block:: smithy
+
+    $version: "2"
+    namespace smithy.example
+
+    @metadata(key: "auditLevel")
+    enum AuditLevel {
+        LOW
+        MEDIUM
+        HIGH
+    }
+
+With that declaration in place, any ``auditLevel`` metadata entry in the
+model is automatically validated against the ``AuditLevel`` shape. For example,
+the following model would produce a validation event because an invalid enum
+value is provided:
+
+.. code-block:: smithy
+
+    $version: "2"
+
+    metadata auditLevel = "unknown"
+
+Building the above model results in the following validation event:
+
+.. code-block::
+
+    ──  ERROR  ──────────────────────────────────────────── TypedMetadata.auditLevel
+    File:  example.smithy:3:23
+
+    3| metadata auditLevel = "unknown"
+     |                       ^
+
+    metadata.auditLevel: String value provided for smithy.example#AuditLevel must
+    be one of the following values: HIGH, LOW, MEDIUM

smithy-model/src/main/java/software/amazon/smithy/model/neighbor/UnreferencedShapes.java

@@ -13,13 +13,19 @@
 import software.amazon.smithy.model.selector.Selector;
 import software.amazon.smithy.model.shapes.Shape;
 import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.model.traits.MetadataTrait;
 import software.amazon.smithy.model.traits.TraitDefinition;
 import software.amazon.smithy.utils.FunctionalUtils;
 
 /**
- * Finds shapes that are not connected to a "root" shape, are not trait definitions, are not referenced by trait
- * definitions, and are not referenced in trait values through
- * {@link software.amazon.smithy.model.traits.IdRefTrait}.
+ * Finds shapes that do not meet any of the following criteria:
+ *
+ * <ul>
+ *     <li>The shape is connected to a "root" shape.
+ *     <li>The shape is a trait definition or is connected to a trait definition.
+ *     <li>The shape is referenced in a trait value through {@link software.amazon.smithy.model.traits.IdRefTrait}.
+ *     <li>The shape is a metadata definition or is connected to a metadata definition.
+ * </ul>
  *
  * <p>The "root" shapes defaults to all service shapes in the model. You can customize this by providing a selector
  * that considers every matching shape a root shape. For example, a model might consider all shapes marked with
@@ -82,6 +88,11 @@ public Set<Shape> compute(Model model) {
             shapeWalker.iterateShapes(trait, traversed).forEachRemaining(shape -> connected.add(shape.getId()));
         }
 
+        // Don't remove shapes that are metadata definitions or connected to metadata definitions.
+        for (Shape metadata : model.getShapesWithTrait(MetadataTrait.class)) {
+            shapeWalker.iterateShapes(metadata, traversed).forEachRemaining(shape -> connected.add(shape.getId()));
+        }
+
         // Any shape that wasn't identified as connected to a root is considered unreferenced.
         Set<Shape> result = new HashSet<>();
         for (Shape shape : model.toSet()) {

smithy-model/src/main/java/software/amazon/smithy/model/traits/MetadataTrait.java

@@ -0,0 +1,92 @@
+/*
+ * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package software.amazon.smithy.model.traits;
+
+import software.amazon.smithy.model.node.Node;
+import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.utils.SmithyBuilder;
+import software.amazon.smithy.utils.ToSmithyBuilder;
+
+/**
+ * Globally declares the type of a metadata key.
+ *
+ * <p>When this trait is applied to a shape, any metadata entry in the model
+ * with the same key is validated against the targeted shape. A given
+ * metadata key may only have its type declared once across the entire model.
+ */
+public final class MetadataTrait extends AbstractTrait implements ToSmithyBuilder<MetadataTrait> {
+    public static final ShapeId ID = ShapeId.from("smithy.api#metadata");
+
+    private final String key;
+
+    private MetadataTrait(Builder builder) {
+        super(ID, builder.getSourceLocation());
+        this.key = SmithyBuilder.requiredState("key", builder.key);
+    }
+
+    /**
+     * @return Creates a builder for a {@link MetadataTrait}.
+     */
+    public static Builder builder() {
+        return new Builder();
+    }
+
+    /**
+     * @return The metadata key that this trait defines a type for.
+     */
+    public String getKey() {
+        return key;
+    }
+
+    @Override
+    public Builder toBuilder() {
+        return builder().sourceLocation(getSourceLocation()).key(key);
+    }
+
+    @Override
+    protected Node createNode() {
+        return Node.objectNodeBuilder()
+                .sourceLocation(getSourceLocation())
+                .withMember("key", key)
+                .build();
+    }
+
+    public static final class Provider extends AbstractTrait.Provider {
+        public Provider() {
+            super(ID);
+        }
+
+        @Override
+        public Trait createTrait(ShapeId target, Node value) {
+            Builder builder = builder().sourceLocation(value.getSourceLocation());
+            builder.key(value.expectObjectNode().expectStringMember("key").getValue());
+            MetadataTrait result = builder.build();
+            result.setNodeCache(value);
+            return result;
+        }
+    }
+
+    public static final class Builder extends AbstractTraitBuilder<MetadataTrait, Builder> {
+        private String key;
+
+        private Builder() {}
+
+        /**
+         * Sets the metadata key that this trait defines a type for.
+         *
+         * @param key The metadata key. Required.
+         * @return Returns the builder.
+         */
+        public Builder key(String key) {
+            this.key = key;
+            return this;
+        }
+
+        @Override
+        public MetadataTrait build() {
+            return new MetadataTrait(this);
+        }
+    }
+}