Implement Typed Documents and TypeRegistry #282

jterapin · 2025-03-06T21:47:57Z

Description: Implementation for Typed Documents and TypeRegistry. Currently only supports JSON documents.

Customer Experience

Customer can instantiate a Document by using standard Ruby objects as its foundation:

string_doc = Smithy::Schema::Document.new('Hello World!')
string_doc.data
=> "Hello World!"

map_doc = Smithy::Schema::Document.new({ :name => 'Hamburger' })
map_doc.data
=> { "name" => "Hamburger" }

Customers can instantiate a Document with a runtime shape and its schema:

# given the following runtime shape:
typed_bird = BirdService::Types::Bird.new(name: 'Pidgey')
=> #<struct BirdService::Types::Bird name="Pidgey">

# provide runtime shape and its schema
bird_doc = Document.new(typed_bird, BirdService::Schema::Bird).data
=> { "name" => "Pidegey" }

When a runtime shape is used to create a Document, we capture the absolute shape id from its schema.

bird_doc.discriminator
=> "example.com#Bird"

Customers can reconstruct the runtime shape of a document by using as_typed method with a shape schema:

bird_doc.as_typed(BirdService::Schema::Bird)
=> #<struct BirdService::Types::Bird name="Pidgey">

Customers can use the TypeRegistry to find schema based on shape id:

BirdService::Schema::TypeRegistry.schema_by_id('somebird')
=> #<BirdService::Schema::Bird>

Remaining ToDos

Add RBS support for public interfaces ONLY
Add more Documentation (maybe)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

gems/smithy-schema/lib/smithy-schema/document.rb

jterapin · 2025-04-14T20:38:47Z

gems/smithy-schema/lib/smithy-schema/document.rb

+          end
+
+          opts = opts.except(:schema)
+          # case 1 - extract data from runtime shape, schema is required to know to properly extract


Note: I'll remove it before merging.

gems/smithy-schema/lib/smithy-schema/type_registry.rb

gems/smithy-schema/spec/smithy-schema/document_spec.rb

gems/smithy-schema/lib/smithy-schema/document.rb

jterapin · 2025-04-15T03:23:13Z

gems/smithy-schema/lib/smithy-schema/document.rb

+module Smithy
+  module Schema
+    # A Smithy document type, representing typed or untyped data from Smithy data model.
+    # ## Document types


I could add more details (e.g. code examples) but I might punt that to the Smithy-Ruby Wiki instead.

I think in-code examples/documentation are preferred but it doesn't have to be done this moment.

Yea - I'd agree examples and more detail would be better as in code documentation.

I will add them now - since information is very fresh in my head.

jterapin · 2025-04-15T03:28:29Z

gems/smithy-schema/lib/smithy-schema/document.rb

+      # @option options [Boolean] :use_timestamp_format Whether to use the `timestampFormat`
+      #  trait or ignore it when creating a {Document} with given schema. The `timestampFormat`
+      #  trait is ignored by default.
+      # @option options [Boolean] :use_json_name Whether to use the `jsonName` trait or ignore
+      #  it when creating a {Document} with given schema. The `jsonName` trait is ignored
+      #  by default.


I was pretty torn on whether I want to have these options around. These options only applies when we create a document with a given schema to decide whether we want to honor specific traits in the @data

My idea is that once @data is set - we can simply do JSON.dump(document.data) to load the data for a request - in lieu of having to iterate over the document data to adhere to traits during SERDE process.

If you are referring to the use_x options, I think they're pretty useless, and can always be added later. I'd start with them removed.

I noticed that there's test cases now, I might see what they ended up doing to determine whether i want to keep these options around.

gems/smithy-schema/lib/smithy-schema/document_utils.rb

projections/shapes/lib/shapes/schema.rb

mullermp

I put some comments to start with, didn't dig into any testing code.

projections/weather/lib/weather/schema.rb

gems/smithy-schema/lib/smithy-schema/type_registry.rb

mullermp · 2025-04-15T15:26:36Z

gems/smithy-schema/lib/smithy-schema/type_registry.rb

+      #  with the given typed shape.
+      # @param [Class] type
+      # @return [Boolean]
+      def schema_by_type?(type)


Where is schema_by_type used or useful? Isn't this also a shape not a schema?

Yeah I'll update the method names to be aligned with the shapes references.

These method will be used in case #2 mentioned in this comment: https://github.com/smithy-lang/smithy-ruby/pull/282/files#r2027627236

When a customer decides to pass a runtime shape for the document param - we need a way to find its shape-id so we can load the discriminator for the service to find it in their type registry

This is a proposed solution instead of having runtime shapes know its own schema.

mullermp · 2025-04-15T15:31:55Z

gems/smithy-client/lib/smithy-client/base.rb

@@ -161,6 +161,16 @@ def service=(service)
          define_operation_methods
        end

+        # @return [Schema::TypeRegistry]
+        def type_registry


What's the intention of placing this on base and why is it needed here? This will add a method to clients. I think we should bias to make this feature for schema gems only for now.

I think all gems should have the type register available - though I'm not sure it belongs as a method on client class - maybe as a method on Schema module, eg: Schema.type_registery

We need access to TypeRegistry during SERDE process - when it comes to deserializing typed documents. I haven't quite figured out where would be a good "home" for this.

I'm thinking of few options...
1 - Update Smithy::Client::HandlerContext to have access to the type registry (when deserializing typed documents - we can find the associated shape) - like: context.type_registry
2 - Keep the reference on the client so we don't need to add another attribute on the HandlerContext - we could do: context.client.type_registry to access during SERDE process
3 - Should ServiceShape have an attribute called type_registry so we could do: client.service.type_registry for easily access?
3 - Just omit the reference for now until we need it. If you need it for whatever reason, you gotta use this constant: BirdService::Schema.TYPE_REGISTRY

If we decide to omit this now, when do we need it?
Any JSON-esque protocol since they support JSON docs

What do y'all think?

gems/smithy-schema/spec/smithy-schema/document_spec.rb

mullermp · 2025-04-15T15:54:00Z

gems/smithy/spec/support/examples/schema_examples.rb

+    let(:typed_shapes) do
+      fixture['shapes'].select do |_k, v|
+        %w[union structure].include?(v['type']) &&
+          !v['traits']&.include?('smithy.api#trait')


It would have any union/structure wouldn't it?

gems/smithy-schema/lib/smithy-schema/document_utils.rb

gems/smithy-schema/lib/smithy-schema/document.rb

mullermp · 2025-04-15T16:04:56Z

gems/smithy-schema/lib/smithy-schema/document.rb

+        return if data.nil?
+
+        case data
+        when Smithy::Schema::Structure


Isn't this already "done"? If it's a structure then it's already typed.

Can you expand on that? what do you mean by being "done" - I still want to re-format the typed shape into a document format.

alextwoods

Nice - looking good overall!

alextwoods · 2025-04-15T20:46:56Z

gems/smithy-client/lib/smithy-client/base.rb

@@ -161,6 +161,16 @@ def service=(service)
          define_operation_methods
        end

+        # @return [Schema::TypeRegistry]
+        def type_registry


I think all gems should have the type register available - though I'm not sure it belongs as a method on client class - maybe as a method on Schema module, eg: Schema.type_registery

alextwoods · 2025-04-15T20:48:19Z

gems/smithy-schema/lib/smithy-schema/document.rb

+module Smithy
+  module Schema
+    # A Smithy document type, representing typed or untyped data from Smithy data model.
+    # ## Document types


Yea - I'd agree examples and more detail would be better as in code documentation.

gems/smithy-schema/lib/smithy-schema/document.rb

alextwoods · 2025-04-15T20:55:45Z

gems/smithy-schema/lib/smithy-schema/document.rb

+        case data
+        when Smithy::Schema::Structure
+          schema = opts[:schema]
+          if schema.nil? || !valid_schema?(schema)


It does still seem weird to me that the run time type doesn't know its own schema

I think it would be useful to have that information at hand but we had some previous discussion on whether this should be a thing or not - based on context of typed shapes being PORO . I don't have a strong opinion but I think starting out without is fine until we have a use case where we absolutely need them to.

gems/smithy-schema/lib/smithy-schema/document_utils.rb

gems/smithy-schema/lib/smithy-schema/type_registry.rb

jterapin

Responding back to comments first before I start revising. Thanks!

jterapin · 2025-04-17T17:17:32Z

gems/smithy-client/lib/smithy-client/base.rb

@@ -161,6 +161,16 @@ def service=(service)
          define_operation_methods
        end

+        # @return [Schema::TypeRegistry]
+        def type_registry


We need access to TypeRegistry during SERDE process - when it comes to deserializing typed documents. I haven't quite figured out where would be a good "home" for this.

I'm thinking of few options...
1 - Update Smithy::Client::HandlerContext to have access to the type registry (when deserializing typed documents - we can find the associated shape) - like: context.type_registry
2 - Keep the reference on the client so we don't need to add another attribute on the HandlerContext - we could do: context.client.type_registry to access during SERDE process
3 - Should ServiceShape have an attribute called type_registry so we could do: client.service.type_registry for easily access?
3 - Just omit the reference for now until we need it. If you need it for whatever reason, you gotta use this constant: BirdService::Schema.TYPE_REGISTRY

If we decide to omit this now, when do we need it?
Any JSON-esque protocol since they support JSON docs

What do y'all think?

jterapin · 2025-04-17T17:18:36Z

gems/smithy-schema/lib/smithy-schema/document.rb

+module Smithy
+  module Schema
+    # A Smithy document type, representing typed or untyped data from Smithy data model.
+    # ## Document types


I will add them now - since information is very fresh in my head.

gems/smithy-schema/lib/smithy-schema/document_utils.rb

gems/smithy-schema/lib/smithy-schema/document.rb

jterapin · 2025-04-17T17:34:21Z

gems/smithy-schema/lib/smithy-schema/document.rb

+      # @option options [Boolean] :use_timestamp_format Whether to use the `timestampFormat`
+      #  trait or ignore it when creating a {Document} with given schema. The `timestampFormat`
+      #  trait is ignored by default.
+      # @option options [Boolean] :use_json_name Whether to use the `jsonName` trait or ignore
+      #  it when creating a {Document} with given schema. The `jsonName` trait is ignored
+      #  by default.


I noticed that there's test cases now, I might see what they ended up doing to determine whether i want to keep these options around.

jterapin · 2025-04-17T18:35:05Z

gems/smithy-schema/lib/smithy-schema/type_registry.rb

+      #  with the given typed shape.
+      # @param [Class] type
+      # @return [Boolean]
+      def schema_by_type?(type)


Yeah I'll update the method names to be aligned with the shapes references.

These method will be used in case #2 mentioned in this comment: https://github.com/smithy-lang/smithy-ruby/pull/282/files#r2027627236

When a customer decides to pass a runtime shape for the document param - we need a way to find its shape-id so we can load the discriminator for the service to find it in their type registry

This is a proposed solution instead of having runtime shapes know its own schema.

gems/smithy-schema/lib/smithy-schema/type_registry.rb

gems/smithy-schema/spec/smithy-schema/document_spec.rb

jterapin added 3 commits March 6, 2025 12:12

Add type registry prototype class

d33fcdf

Add type registry to codegenerated schema

5ff40cc

Update projections

66a6285

jterapin changed the title ~~Implement Typed Documents and TypeRegistry~~ [WIP] Implement Typed Documents and TypeRegistry Mar 6, 2025

jterapin added 24 commits March 7, 2025 13:39

Merge branch 'decaf' into typed_documents

c5e45ed

Merge branch 'decaf' into typed_documents

2bd15a2

Merge branch 'decaf' into typed_documents

03bfa82

Update requires

e6435d5

Add initial document implementation

0830827

Merge decaf into branch

877654f

Update to include cbor

a61318f

Expand on typed docs

4edfae3

Update file names

ff959f1

Merge branch 'decaf' into typed_documents

8b9b560

More refactoring

598db66

Merge branch 'decaf' into typed_documents

3a4c0d1

Remove scratches

269b2b5

Fix rubocop

90c58ce

Clean up document

a1e46cc

Clean document specs

8b666cd

Update TypeRegistry

2ddf4bd

Add documentation

112ddf4

Add TypeRegistry specs

6283813

Merge branch 'decaf' into typed_documents

efbfa5e

Add TypeRegistry tests

88ff845

Update projections

66b2cde

Update syntax

22998a0

Update projections

9afeacd

jterapin changed the title ~~[WIP] Implement Typed Documents and TypeRegistry~~ Implement Typed Documents and TypeRegistry Apr 15, 2025

jterapin commented Apr 15, 2025

View reviewed changes

jterapin marked this pull request as ready for review April 15, 2025 03:48

jterapin requested a review from mullermp April 15, 2025 03:49

mullermp reviewed Apr 15, 2025

View reviewed changes

alextwoods reviewed Apr 15, 2025

View reviewed changes

Merge branch 'decaf' into typed_documents

232844c

jterapin commented Apr 17, 2025

View reviewed changes

jterapin added 10 commits April 17, 2025 12:58

Change schema name to shapes to stay aligned

e8920fd

Remove as_typed method from TypeRegistry

ef8c027

Create TimeHelper module

3854661

Fix timestamp failures

48e1b0f

Refactor type registry per feedbacks

04d0d5b

Update 1 projection

b35e09b

Update weather projection

66825be

Use SchemaHelper for testing

4efe669

Update TypeRegistry to use SchemaHelper for testing

73e0e83

Add TypeRegistry documentation

60ef29c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Typed Documents and TypeRegistry #282

Implement Typed Documents and TypeRegistry #282

jterapin commented Mar 6, 2025 •

edited

Loading

jterapin Apr 14, 2025

jterapin Apr 15, 2025

mullermp Apr 15, 2025

alextwoods Apr 15, 2025

jterapin Apr 17, 2025

jterapin Apr 15, 2025

mullermp Apr 15, 2025

jterapin Apr 17, 2025

mullermp left a comment

mullermp Apr 15, 2025

jterapin Apr 17, 2025

mullermp Apr 15, 2025

alextwoods Apr 15, 2025

jterapin Apr 17, 2025

mullermp Apr 15, 2025

mullermp Apr 15, 2025

jterapin Apr 17, 2025

alextwoods left a comment

alextwoods Apr 15, 2025

alextwoods Apr 15, 2025

alextwoods Apr 15, 2025

jterapin Apr 17, 2025

jterapin left a comment

jterapin Apr 17, 2025

jterapin Apr 17, 2025

jterapin Apr 17, 2025

jterapin Apr 17, 2025

Implement Typed Documents and TypeRegistry #282

Are you sure you want to change the base?

Implement Typed Documents and TypeRegistry #282

Conversation

jterapin commented Mar 6, 2025 • edited Loading

Customer Experience

Remaining ToDos

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mullermp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alextwoods left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jterapin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jterapin commented Mar 6, 2025 •

edited

Loading