Pass empty JSON arrays to gRPC as empty slices instead of nil

[zhangcz828]

I'm encountering the same problem described in #1673 and would like to provide more context and a concrete proposal.

Problem:
Given a proto like:

syntax = "proto3";

package example;

import "google/protobuf/empty.proto";
import "google/api/annotations.proto";

option go_package = "examplepb";

// Request message with a repeated string field
message MyRequest {
  repeated string items = 1;
}

// Service definition with HTTP annotation
service MyService {
  rpc SendItems(MyRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      patch: "/v1/items"
      body: "*"
    };
  }
}

When sending a request with an empty array in JSON (e.g., curl -X PATCH -d '{"items":[]}' localhost:7777/v1/items), the generated Go gRPC handler receives nil for the items field, not an empty slice. This is not due to sending null, but specifically an empty array.

Constraints:

• Wrapping the field in a message is not an option, as the proto definitions are already published and used for OpenAPI generation.
• Customizing the gateway marshaler does not help, since the protobuf encoding treats both nil and empty slices as nil in the generated Go code, so the handler cannot distinguish between omitted and explicitly empty arrays.

Proposal:

1. Use a custom runtime.WithMetadata option in grpc-gateway. When the HTTP request reaches the gateway, parse the body and record the paths of empty array fields (including nested fields) in the context metadata.

   • Example implementation: https://gist.github.com/zhangcz828/e1606cd5dadf1ae175c5cacd3fab4b44

2. Add a gRPC interceptor that reads the x-empty-fields metadata, and uses reflection to set the corresponding fields in the request struct to their type's empty value (e.g., set a string slice to []string{}).

   • Supports nested field paths (e.g., "foo.bar.items").
   • Handles mapping between JSON field names and Go struct fields generated from proto (case-insensitive, supports snake_case and camelCase).
   • Example implementation: https://gist.github.com/zhangcz828/0a8f8639b9ff0d376c919b7ef4da6d3a

Question:
Would this approach to handling "nullable" or explicitly empty repeated fields be acceptable for inclusion in this repo, so that other projects can benefit from it? Feedback and suggestions are welcome.

[johanbrandhorst]

Hi, thanks for your issue! As I said in #1673, I'm sympathetic to the problem, but I don't think this is the responsibility of the gateway, it is the responsibility of the JSON unmarshaler, which is already configurable. You may be able to write a custom marshaler that distinguishes between {"items":[]} and {"items":null} and sets your slice to a zero-length instance or null respectively, but I don't think we should build any workarounds into the gateway for this. Have you looked at wrapping the protojson marshaler?

[zhangcz828]

Hi @johanbrandhorst, sorry for any confusion earlier, let me clarify the situation.

When we send a request like curl -X PATCH -d '{}' localhost:7777/v1/items (with no data), the generated Go gRPC handler receives Items as nil.
Likewise, if we send curl -X PATCH -d '{"items":[]}' localhost:7777/v1/items, the handler also receives Items as nil.

We attempted to address this by implementing a custom marshaler, but that did not resolve the issue. Upon further investigation, we found that this behavior is inherent to how protobuf encoding works: regardless of the marshaler used in grpc-gateway, both requests result in the proto message being encoded without the items field when transferring data from the grpc client to the grpc server. In other words, protobuf does not encode nil or empty values, so both cases are treated as nil during the transformation, and the gRPC server always receives an empty (nil) value. The marshaler configuration in grpc-gateway cannot override this protobuf encoding behavior.

Given these limitations, we are considering extracting information about empty arrays into context metadata within the transcoder, and then using a gRPC server interceptor to distinguish between a truly missing field and an explicitly empty array. This would allow us to set the value to an empty slice instead of nil when appropriate.

If you have any suggestions or see a better approach, please let us know. Thanks!

[johanbrandhorst]

I see, thank you for the clarification, that is unfortunate. It's not great, but have you tried using protobuf.struct.ListValue for the arrays you care about? It might let to distinguish between null and [] but you lose other type safety. Your custom metadata parser and context smuggler also works obviously but I don't want to build that into the repository as it's a bit of a hack and requires buffering the entire request body. I think it might still be the best solution for yourself, but I don't think it's a common enough problem for users that we need to include it in the repo. Thanks for the suggestion though!

[zhangcz828]

@johanbrandhorst thanks for your reply!