Map DateTime to google.protobuf.Timestamp and document temporal string formats

`DateTime` fields now use the well-known `google.protobuf.Timestamp` type
(with `google/protobuf/timestamp.proto` imported automatically) instead of
`string`. A Timestamp cannot be malformed the way a string can, and proto
consumers get language-native timestamp types. Note that a Timestamp is a
UTC instant, so a publisher's original UTC offset is not preserved;
`t.protobuf type: "string"` remains available to override.

To support this, `t.protobuf` gains two options usable by any scalar:

- `import:` maps a scalar to an externally defined proto type, emitting the
  needed `import` statement in `schema.proto`.
- `comment:` documents the expected format on each generated field, used by
  the built-in `string`-typed temporal scalars (`Date`, `LocalTime`,
  `TimeZone`) whose proto type is wider than the ElasticGraph type (e.g.
  `// ISO 8601 date`). Values are still validated at ingestion time, just
  as with JSON ingestion.

Author: jwils

elasticgraph-proto_ingestion/README.md

@@ -95,6 +95,22 @@ ElasticGraph.define_schema do |schema|
 end
 ```
 
+A custom scalar can also map to an externally defined proto type by passing `import:` with the
+proto file that defines it, and `comment:` documents the expected format on each generated field
+(useful when the proto type is wider than the ElasticGraph type):
+
+```ruby
+# in config/schema/phone_number.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "PhoneNumber" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string"
+    t.protobuf type: "string", comment: "E.164 phone number"
+  end
+end
+```
+
 ### Stable Field Numbers
 
 `schema_artifacts:dump` automatically reads and writes `proto_field_numbers.yaml`
@@ -138,23 +154,30 @@ enums:
 
 The generated `schema.proto` uses these built-in scalar mappings:
 
-| ElasticGraph Type | Protobuf Type |
-|-------------------|------------|
-| `Boolean`         | `bool`     |
-| `Cursor`          | `string`   |
-| `Date`            | `string`   |
-| `DateTime`        | `string`   |
-| `Float`           | `double`   |
-| `ID`              | `string`   |
-| `Int`             | `int32`    |
-| `JsonSafeLong`    | `int64`    |
-| `LocalTime`       | `string`   |
-| `LongString`      | `int64`    |
-| `String`          | `string`   |
-| `TimeZone`        | `string`   |
-| `Untyped`         | `string`   |
+| ElasticGraph Type | Protobuf Type               |
+|-------------------|-----------------------------|
+| `Boolean`         | `bool`                      |
+| `Cursor`          | `string`                    |
+| `Date`            | `string`                    |
+| `DateTime`        | `google.protobuf.Timestamp` |
+| `Float`           | `double`                    |
+| `ID`              | `string`                    |
+| `Int`             | `int32`                     |
+| `JsonSafeLong`    | `int64`                     |
+| `LocalTime`       | `string`                    |
+| `LongString`      | `int64`                     |
+| `String`          | `string`                    |
+| `TimeZone`        | `string`                    |
+| `Untyped`         | `string`                    |
 
 Additionally:
+- `DateTime` uses the [well-known `Timestamp` type](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp);
+  `schema.proto` imports `google/protobuf/timestamp.proto` automatically. Note that a `Timestamp`
+  is a UTC instant, so a publisher's original UTC offset is not preserved.
+- `string`-typed temporal scalars (`Date`, `LocalTime`, `TimeZone`) are wider than the
+  ElasticGraph types they carry, so generated fields of these types document the expected format
+  in a comment (e.g. `// ISO 8601 date, e.g. "2024-11-25"`). Values are validated when events
+  are ingested, just as with JSON ingestion.
 - List types become `repeated` fields.
 - Lists of lists (e.g. `[[Float!]!]!`) are not supported because Protocol Buffers cannot represent
   them directly. Schema artifact generation raises an error identifying the unsupported field.

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/api_extension.rb

@@ -21,21 +21,21 @@ module SchemaDefinition
       # Module designed to be extended onto an {ElasticGraph::SchemaDefinition::API} instance
       # to enable protobuf schema artifact generation.
       module APIExtension
-        # Maps built-in ElasticGraph scalar types to proto field types.
-        PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE = {
-          "Boolean" => "bool",
-          "Cursor" => "string",
-          "Date" => "string",
-          "DateTime" => "string",
-          "Float" => "double",
-          "ID" => "string",
-          "Int" => "int32",
-          "JsonSafeLong" => "int64",
-          "LocalTime" => "string",
-          "LongString" => "int64",
-          "String" => "string",
-          "TimeZone" => "string",
-          "Untyped" => "string"
+        # Maps built-in ElasticGraph scalar types to their `protobuf` configuration options.
+        PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE = {
+          "Boolean" => {type: "bool"},
+          "Cursor" => {type: "string"},
+          "Date" => {type: "string", comment: %(ISO 8601 date, e.g. "2024-11-25")},
+          "DateTime" => {type: "google.protobuf.Timestamp", import: "google/protobuf/timestamp.proto"},
+          "Float" => {type: "double"},
+          "ID" => {type: "string"},
+          "Int" => {type: "int32"},
+          "JsonSafeLong" => {type: "int64"},
+          "LocalTime" => {type: "string", comment: %(ISO 8601 local time, e.g. "14:23:12")},
+          "LongString" => {type: "int64"},
+          "String" => {type: "string"},
+          "TimeZone" => {type: "string", comment: %(IANA time zone identifier, e.g. "America/Los_Angeles")},
+          "Untyped" => {type: "string"}
         }.freeze
 
         # Wires up the protobuf extensions when this module is extended onto an API instance.
@@ -51,7 +51,8 @@ def self.extended(api)
             if type.is_a?(SchemaElements::ScalarTypeExtension)
               # Use the reverted (original) type name so that built-in scalars renamed via
               # `type_name_overrides` still resolve to their proto types.
-              type.protobuf type: PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE.fetch(type.type_ref.with_reverted_override.name)
+              options = PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE.fetch(type.type_ref.with_reverted_override.name)
+              type.protobuf(type: options.fetch(:type), import: options[:import], comment: options[:comment])
             end
           end
         end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/schema.rb

@@ -46,6 +46,7 @@ def initialize(
           @proto_enum_value_numbers_by_enum = normalize_proto_enum_value_number_mappings(proto_field_number_mappings)
           @used_field_numbers_by_message = {}
           @used_enum_value_numbers_by_enum = {}
+          @imports = ::Set.new
           @definitions_by_name = {}
           @definition_kind_by_name = {}
           @type_name_by_proto_name = {}
@@ -63,6 +64,7 @@ def to_proto
           sections = [
             %(syntax = "proto3";),
             "package #{@package_name};",
+            *render_imports,
             render_definitions
           ]
 
@@ -138,7 +140,9 @@ def proto_field_type_for(type_ref, context_type_name:, context_field_name:)
             raise Errors::SchemaError, "Type `#{base_type_ref.unwrapped_name}` cannot be resolved for proto generation."
           end
 
-          [list_depth == 1, register_type(resolved)]
+          base_type_name = register_type(resolved)
+          base_type_comment = (SchemaElements::ScalarTypeExtension === resolved) ? resolved.protobuf_comment : nil
+          [list_depth == 1, base_type_name, base_type_comment]
         end
 
         # Returns the stable protobuf number for a message field.
@@ -189,6 +193,13 @@ def enum_value_numbers_for(enum_name, value_names)
           end
         end
 
+        # Registers a proto file import required by a referenced external type.
+        #
+        # @api private
+        def register_import(import)
+          @imports << import if import
+        end
+
         private
 
         def indexed_types
@@ -270,6 +281,11 @@ def render_definitions
             .join("\n\n")
         end
 
+        def render_imports
+          return [] if @imports.empty?
+          [@imports.sort.map { |import| %(import "#{import}";) }.join("\n")]
+        end
+
         def normalize_proto_field_number_mappings(raw_mappings)
           return {} if raw_mappings.nil?
           unless raw_mappings.is_a?(Hash)

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/schema_elements/object_interface_and_union_extension.rb

@@ -47,7 +47,7 @@ def render_proto_message(schema, message_name)
             else
               fields.each do |field|
                 field_name = Identifier.field_name(field.name)
-                repeated, field_type = schema.proto_field_type_for(field.type, context_type_name: name, context_field_name: field.name)
+                repeated, field_type, type_comment = schema.proto_field_type_for(field.type, context_type_name: name, context_field_name: field.name)
                 field_number = schema.field_number_for(
                   message_name: message_name,
                   type_name: name,
@@ -56,7 +56,10 @@ def render_proto_message(schema, message_name)
                 )
                 label = "repeated " if repeated
                 line = "  #{label}#{field_type} #{field_name} = #{field_number};"
-                line += " // source name: #{field.name}" if field_name != field.name
+                comment_parts = [] # : ::Array[::String]
+                comment_parts << "source name: #{field.name}" if field_name != field.name
+                comment_parts << type_comment if type_comment
+                line += " // #{comment_parts.join("; ")}" if comment_parts.any?
                 lines << line
               end
             end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/schema_elements/scalar_type_extension.rb

@@ -18,19 +18,36 @@ module ScalarTypeExtension
           # @dynamic protobuf_type
           attr_reader :protobuf_type
 
-          # Configures the protobuf type for this scalar type.
+          # Proto file to import for the configured protobuf type, if it is an externally defined type.
+          # @dynamic protobuf_import
+          attr_reader :protobuf_import
+
+          # Comment rendered on generated proto fields of this scalar type (e.g. to document a string format).
+          # @dynamic protobuf_comment
+          attr_reader :protobuf_comment
+
+          # Configures the protobuf type for this scalar type. In addition to proto scalar types,
+          # an externally defined message type can be used by passing `import:` with the proto file
+          # that defines it (e.g. a well-known type such as `google.protobuf.Timestamp`). When the
+          # proto type is wider than the ElasticGraph type (such as a `string`-typed `Date`),
+          # `comment:` documents the expected format on each generated field.
           #
-          # @param type [String] protobuf scalar type name
+          # @param type [String] protobuf type name
+          # @param import [String, nil] proto file to import for the type, when it is not a proto scalar
+          # @param comment [String, nil] comment rendered on generated fields of this type
           # @return [void]
-          def protobuf(type:)
+          def protobuf(type:, import: nil, comment: nil)
             @protobuf_type = type
+            @protobuf_import = import
+            @protobuf_comment = comment
           end
 
           # Returns this scalar's proto field type.
           #
           # @return [String]
           # @raise [Errors::SchemaError] when missing
-          def to_proto(_schema = nil)
+          def to_proto(schema = nil)
+            schema&.register_import(protobuf_import)
             protobuf_type ||
               raise(Errors::SchemaError, "Protobuf type not configured for scalar type `#{name}`. " \
                 'To proceed, call `protobuf type: "TYPE"` on the scalar type definition.')

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/api_extension.rbs

@@ -2,7 +2,7 @@ module ElasticGraph
   module ProtoIngestion
     module SchemaDefinition
       module APIExtension: ::ElasticGraph::SchemaDefinition::API
-        PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE: ::Hash[::String, ::String]
+        PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE: ::Hash[::String, ::Hash[::Symbol, ::String]]
 
         def self.extended: (::ElasticGraph::SchemaDefinition::API & APIExtension) -> void
         def proto_schema_artifacts: (?package_name: ::String) -> void

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/schema.rbs

@@ -21,6 +21,7 @@ module ElasticGraph
         @proto_enum_value_numbers_by_enum: ::Hash[::String, ::Hash[::String, ::Integer]]
         @used_field_numbers_by_message: ::Hash[::String, ::Set[::Integer]]
         @used_enum_value_numbers_by_enum: ::Hash[::String, ::Set[::Integer]]
+        @imports: ::Set[::String]
         @definitions_by_name: ::Hash[::String, ::String?]
         @definition_kind_by_name: ::Hash[::String, ::Symbol]
         @type_name_by_proto_name: ::Hash[::String, ::String]
@@ -42,11 +43,12 @@ module ElasticGraph
           name_in_index: ::String
         ) -> ::Integer
         def enum_value_numbers_for: (::String, ::Array[::String]) -> ::Hash[::String, ::Integer]
+        def register_import: (::String?) -> void
         def proto_field_type_for: (
           ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference,
           context_type_name: ::String,
           context_field_name: ::String
-        ) -> [bool, ::String]
+        ) -> [bool, ::String, ::String?]
 
         private
 
@@ -65,6 +67,7 @@ module ElasticGraph
           ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference
         ) -> [::Integer, ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference]
         def render_definitions: () -> ::String
+        def render_imports: () -> ::Array[::String]
         def normalize_proto_field_number_mappings: (untyped) -> ::Hash[::String, fieldNumberMappingsByFieldName]
         def normalize_proto_enum_value_number_mappings: (untyped) -> ::Hash[::String, ::Hash[::String, ::Integer]]
         def normalize_field_number_mapping_entry: (::String, ::String, untyped) -> [::Integer, ::String]

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/schema_elements/scalar_type_extension.rbs

@@ -4,8 +4,10 @@ module ElasticGraph
       module SchemaElements
         module ScalarTypeExtension: ::ElasticGraph::SchemaDefinition::SchemaElements::ScalarType
           attr_reader protobuf_type: ::String?
+          attr_reader protobuf_import: ::String?
+          attr_reader protobuf_comment: ::String?
 
-          def protobuf: (type: ::String) -> void
+          def protobuf: (type: ::String, ?import: ::String?, ?comment: ::String?) -> void
           def to_proto_field_type: () -> ::String
           def to_proto: (?Schema?) -> ::String
         end

elasticgraph-proto_ingestion/spec/unit/elastic_graph/proto_ingestion/schema_definition/api_extension_spec.rb

@@ -29,11 +29,12 @@ module SchemaDefinition
 
           expect(proto_schema_from(results)).to include(
             "package sales.v1;",
+            'import "google/protobuf/timestamp.proto";',
             "string id = 1;",
             "int32 count = 2;",
             "double cost = 3;",
             "bool active = 4;",
-            "string created_at = 5;",
+            "google.protobuf.Timestamp created_at = 5;",
             "int64 size_bytes = 6;"
           )
         end

elasticgraph-proto_ingestion/spec/unit/elastic_graph/proto_ingestion/schema_definition/schema_spec.rb

@@ -135,6 +135,78 @@ module SchemaDefinition
           expect(proto_schema_from(results)).to include("fixed64 occurred_at = 2;")
         end
 
+        it "maps `DateTime` fields to `google.protobuf.Timestamp`, importing its proto file once" do
+          results = define_proto_schema do |s|
+            s.object_type "Event" do |t|
+              t.field "id", "ID"
+              t.field "created_at", "DateTime"
+              t.field "updated_at", "DateTime"
+              t.index "events"
+            end
+          end
+
+          expect(proto_schema_from(results)).to eq(<<~PROTO)
+            syntax = "proto3";
+
+            package elasticgraph;
+
+            import "google/protobuf/timestamp.proto";
+
+            message Event {
+              string id = 1;
+              google.protobuf.Timestamp created_at = 2;
+              google.protobuf.Timestamp updated_at = 3;
+            }
+          PROTO
+        end
+
+        it "renders format comments on fields whose scalar type documents one" do
+          results = define_proto_schema do |s|
+            s.object_type "Person" do |t|
+              t.field "id", "ID"
+              t.field "birth_date", "Date"
+              t.index "people"
+            end
+          end
+
+          expect(proto_schema_from(results)).to include(
+            %(string birth_date = 2; // ISO 8601 date, e.g. "2024-11-25")
+          )
+        end
+
+        it "combines source name and format comments on a single field" do
+          results = define_proto_schema do |s|
+            s.object_type "Person" do |t|
+              t.field "id", "ID"
+              t.field "option", "Date"
+              t.index "people"
+            end
+          end
+
+          expect(proto_schema_from(results)).to include(
+            %(string option_ = 2; // source name: option; ISO 8601 date, e.g. "2024-11-25")
+          )
+        end
+
+        it "supports `import:` and `comment:` on custom scalar types" do
+          results = define_proto_schema do |s|
+            s.scalar_type "Money" do |t|
+              t.mapping type: "keyword"
+              t.protobuf type: "myapp.types.Money", import: "myapp/types/money.proto", comment: "amount + currency"
+            end
+
+            s.object_type "Order" do |t|
+              t.field "id", "ID"
+              t.field "total", "Money"
+              t.index "orders"
+            end
+          end
+
+          generated = proto_schema_from(results)
+          expect(generated).to include('import "myapp/types/money.proto";')
+          expect(generated).to include("myapp.types.Money total = 2; // amount + currency")
+        end
+
         it "can assign field numbers from configured mappings" do
           results = define_proto_schema do |s|
             s.configure_proto_field_number_mappings(