Add trait to declare types for metadata

.changes/next-release/feature-5c018acaef9147e7499e70c6d8919d6cf0b496c5.json

@@ -0,0 +1,5 @@
+{
+  "type": "feature",
+  "description": "Added a new `metadata` trait that allows model authors to declare types for metadata keys that will be automatically validated when building models.",
+  "pull_requests": []
+}

designs/typed-metadata.md

@@ -0,0 +1,172 @@
+# Typed Metadata
+
+Metadata is a schema-less extensibility mechanism used to associate metadata to
+an entire model. For example, metadata is used to define validators and
+model-wide suppressions.
+
+This document describes a way to define typing information for metadata that is
+automatically validated by Smithy.
+
+## Motivation
+
+The schema-less nature of metadata has allowed it to be used for any purpose
+without much hassle. However, that has come at the cost of increased validation
+complexity. Any tool, including Smithy itself, that uses metadata in a
+structured way has to perform validation itself. By allowing opt-in validation,
+we can centralize and deduplicate that effort.
+
+## Proposal
+
+Model authors may globally declare the type of a metadata key by targeting a
+shape with the `@metadata` trait.
+
+```smithy
+/// Defines a type for a metadata key.
+///
+/// If a matching key is defined in the model, its value will be validated
+/// according to the targeted shape.
+///
+/// The type for any metadata key MUST only be defined once.
+@trait(selector: "dataType :not([trait|input]) :not([trait|output])")
+structure metadata {
+    /// The metadata key to validate. Each key MUST only be defined once.
+    @required
+    @length(min: 1)
+    key: String
+}
+```
+
+For example, the
+[suppressions metadata](https://smithy.io/2.0/spec/model-validation.html#suppressions-metadata)
+could be defined as:
+
+```smithy
+$version: "2.0"
+
+namespace smithy.api
+
+@metadata(key: "suppressions")
+list MetadataSuppressions {
+    member: MetadataSuppression
+}
+
+structure MetadataSuppression {
+    @required
+    id: String
+
+    @required
+    @pattern("^(\*|[_a-zA-Z]\w*(\.[_a-zA-Z]\w*)*)$")
+    namespace: String
+
+    reason: String
+}
+```
+
+### Validation
+
+A metadata key MUST NOT be defined more than once with the `@metadata` trait. If
+a metadata key is defined more than once with the `@metadata` trait, an
+`ERROR`-level validation event will be emitted.
+
+Validation of metadata values will behave no differently than validation
+anywhere else, with severity levels being determined by each individual
+validator.
+
+Shapes targeted by the metadata shape and shapes transitively referenced by
+those shapes will not trigger the unreferenced shapes validator.
+
+Adding or removing this trait is considered backwards-compatible because
+metadata does not inherently change any interface. It may cause a build to break
+if the new validation rejects existing metadata, but that is intended behavior.
+
+## Alternatives
+
+### Inline definitions
+
+The metadata trait globally defines the shape of a metadata key, but we could
+allow local, inline definitions additionally or instead. This can be achieved by
+using a special `$type` key in metadata's node value.
+
+```smithy
+$version: "2.0"
+
+metadata foo = {
+    "$type": "com.example#Foo"
+    "bar": "baz"
+}
+
+namespace com.example
+
+structure Foo {
+    bar: String
+}
+```
+
+The problem with this strategy is that it can only be applied to structure and
+union shapes, because other shape types have nowhere to add the `$type` key or
+it would potentially shadow a valid value key.
+
+Other shapes could instead be defined via a `__type__` metadata key:
+
+```smithy
+$version: "2.0"
+
+metadata "__type__": [
+    ["foo", "com.example#Foo"]
+]
+
+metadata foo = "bar"
+
+namespace com.example
+
+string Foo
+
+@metadata(key: "__type__")
+list MetdataTypeDeclarations {
+    member: MetadataTypeDeclaration
+}
+
+@length(min: 2, max: 2)
+list MetadataTypeDeclaration {
+    member: String
+}
+```
+
+Unfortunately, this is inherently a global declaration that is not inline with
+the value, defeating the intent of an inline definition. The metadata trait is
+better suited to this purpose, as it is clearer, easier to validate, more easily
+discoverable, and more easily trackable.
+
+Smithy 2.1 could introduce new syntax to declare inline metadata types that isn't
+tied to the value, but the AST could not support it without either exposing the
+same problem or making a breaking change.
+
+## FAQ
+
+### Does this interact with metadata merging semantics?
+
+When a given metadata key is defined more than once, it usually results in a
+conflict that fails the build. If both definitions are arrays, however, they are
+instead merged. If a metadata key is defined as a list, the merged array will be
+validated.
+
+This trait does not change merging semantics.
+
+### Will the existing defined metadata keys get metadata-trait-based definitions?
+
+Smithy currently has definitions for three metadata keys: `severityOverrides`,
+`suppressions`, and `validators`. All three could be defined with the metadata
+trait.
+
+However, these three keys are currently parsed and validated early on when
+loading a model, before validation generally is run. They will need to be
+special-cased to either run early or to be skipped in favor of existing
+validation.
+
+These keys will initially be reserved so they can't be defined. As the keys get
+full definitions, their reserve list entries will be removed.
+
+### What happens if a defined metadata key is not used?
+
+Nothing. Metadata keys are inherently optional. A `required` member may be added
+later to enforce presence if there is a need.

docs/source-2.0/spec/model-validation.rst

@@ -867,3 +867,86 @@ Validator definition
       - ``string``
       - The :ref:`severity <severity-definition>` to use when an
         incompatible shape is found. Defaults to ``ERROR`` if not set.
+
+
+.. smithy-trait:: smithy.api#metadata
+.. _metadata-trait:
+
+------------------
+``metadata`` trait
+------------------
+
+:ref:`Metadata <metadata>` is schema-less by default. Any tool that consumes
+metadata, including Smithy itself, must validate it independently. The
+``metadata`` trait lets model authors define a type for a metadata key.
+When the type for a metadata key is defined this way, Smithy will automatically
+validate any value for that metadata key against its defined type.
+
+Summary
+    Defines a type for a metadata key by targeting a shape. When a metadata
+    entry with a matching key is defined in the model, its value is
+    validated against the targeted shape.
+Trait selector
+    ``dataType :not([trait|input]) :not([trait|output])``
+
+    *A :ref:`simple shape <simple-types>` or
+    :ref:`aggregate shape <aggregate-types>` that is not marked with the
+    :ref:`input trait <input-trait>` or :ref:`output trait <output-trait>`.*
+Value type
+    ``structure``
+
+The ``metadata`` trait is a structure that contains the following members:
+
+.. list-table::
+    :header-rows: 1
+    :widths: 10 10 80
+
+    * - Property
+      - Type
+      - Description
+    * - key
+      - ``string``
+      - **Required**. The metadata key whose type is being defined. The
+        type of any metadata key may only be defined once across the entire
+        model. This value must be non-empty.
+
+.. rubric:: Example
+
+Consider a service that configures an ``auditLevel`` metadata entry. Defining
+a type for the key gives Smithy enough information to validate each usage:
+
+.. code-block:: smithy
+
+    $version: "2"
+    namespace smithy.example
+
+    @metadata(key: "auditLevel")
+    enum AuditLevel {
+        LOW
+        MEDIUM
+        HIGH
+    }
+
+With that declaration in place, any ``auditLevel`` metadata entry in the
+model is automatically validated against the ``AuditLevel`` shape. For example,
+the following model would produce a validation event because an invalid enum
+value is provided:
+
+.. code-block:: smithy
+
+    $version: "2"
+
+    metadata auditLevel = "unknown"
+
+Building the above model results in the following validation event:
+
+.. code-block::
+
+    ──  ERROR  ──────────────────────────────────────────── TypedMetadata.auditLevel
+    File:  example.smithy:3:23
+
+    3| metadata auditLevel = "unknown"
+     |                       ^
+
+    metadata.auditLevel: String value provided for smithy.example#AuditLevel must
+    be one of the following values: HIGH, LOW, MEDIUM

smithy-model/src/main/java/software/amazon/smithy/model/neighbor/UnreferencedShapes.java

@@ -13,13 +13,19 @@
 import software.amazon.smithy.model.selector.Selector;
 import software.amazon.smithy.model.shapes.Shape;
 import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.model.traits.MetadataTrait;
 import software.amazon.smithy.model.traits.TraitDefinition;
 import software.amazon.smithy.utils.FunctionalUtils;
 
 /**
- * Finds shapes that are not connected to a "root" shape, are not trait definitions, are not referenced by trait
- * definitions, and are not referenced in trait values through
- * {@link software.amazon.smithy.model.traits.IdRefTrait}.
+ * Finds shapes that do not meet any of the following criteria:
+ *
+ * <ul>
+ *     <li>The shape is connected to a "root" shape.
+ *     <li>The shape is a trait definition or is connected to a trait definition.
+ *     <li>The shape is referenced in a trait value through {@link software.amazon.smithy.model.traits.IdRefTrait}.
+ *     <li>The shape is a metadata definition or is connected to a metadata definition.
+ * </ul>
  *
  * <p>The "root" shapes defaults to all service shapes in the model. You can customize this by providing a selector
  * that considers every matching shape a root shape. For example, a model might consider all shapes marked with
@@ -82,6 +88,11 @@ public Set<Shape> compute(Model model) {
             shapeWalker.iterateShapes(trait, traversed).forEachRemaining(shape -> connected.add(shape.getId()));
         }
 
+        // Don't remove shapes that are metadata definitions or connected to metadata definitions.
+        for (Shape metadata : model.getShapesWithTrait(MetadataTrait.class)) {
+            shapeWalker.iterateShapes(metadata, traversed).forEachRemaining(shape -> connected.add(shape.getId()));
+        }
+
         // Any shape that wasn't identified as connected to a root is considered unreferenced.
         Set<Shape> result = new HashSet<>();
         for (Shape shape : model.toSet()) {

smithy-model/src/main/java/software/amazon/smithy/model/traits/MetadataTrait.java

@@ -0,0 +1,92 @@
+/*
+ * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package software.amazon.smithy.model.traits;
+
+import software.amazon.smithy.model.node.Node;
+import software.amazon.smithy.model.shapes.ShapeId;
+import software.amazon.smithy.utils.SmithyBuilder;
+import software.amazon.smithy.utils.ToSmithyBuilder;
+
+/**
+ * Globally declares the type of a metadata key.
+ *
+ * <p>When this trait is applied to a shape, any metadata entry in the model
+ * with the same key is validated against the targeted shape. A given
+ * metadata key may only have its type declared once across the entire model.
+ */
+public final class MetadataTrait extends AbstractTrait implements ToSmithyBuilder<MetadataTrait> {
+    public static final ShapeId ID = ShapeId.from("smithy.api#metadata");
+
+    private final String key;
+
+    private MetadataTrait(Builder builder) {
+        super(ID, builder.getSourceLocation());
+        this.key = SmithyBuilder.requiredState("key", builder.key);
+    }
+
+    /**
+     * @return Creates a builder for a {@link MetadataTrait}.
+     */
+    public static Builder builder() {
+        return new Builder();
+    }
+
+    /**
+     * @return The metadata key that this trait defines a type for.
+     */
+    public String getKey() {
+        return key;
+    }
+
+    @Override
+    public Builder toBuilder() {
+        return builder().sourceLocation(getSourceLocation()).key(key);
+    }
+
+    @Override
+    protected Node createNode() {
+        return Node.objectNodeBuilder()
+                .sourceLocation(getSourceLocation())
+                .withMember("key", key)
+                .build();
+    }
+
+    public static final class Provider extends AbstractTrait.Provider {
+        public Provider() {
+            super(ID);
+        }
+
+        @Override
+        public Trait createTrait(ShapeId target, Node value) {
+            Builder builder = builder().sourceLocation(value.getSourceLocation());
+            builder.key(value.expectObjectNode().expectStringMember("key").getValue());
+            MetadataTrait result = builder.build();
+            result.setNodeCache(value);
+            return result;
+        }
+    }
+
+    public static final class Builder extends AbstractTraitBuilder<MetadataTrait, Builder> {
+        private String key;
+
+        private Builder() {}
+
+        /**
+         * Sets the metadata key that this trait defines a type for.
+         *
+         * @param key The metadata key. Required.
+         * @return Returns the builder.
+         */
+        public Builder key(String key) {
+            this.key = key;
+            return this;
+        }
+
+        @Override
+        public MetadataTrait build() {
+            return new MetadataTrait(this);
+        }
+    }
+}