Note about HTTP/1.1 roundtrip reduction via array-based auto-batching

[sungam3r]

Is it worth mentioning batch requests? In this case, an array of objects is transmitted via http.

[benjie]

I wondered this too; but I think batch requests will be superseded by HTTP2 so I'm not sure how much value there is in it. That said, it's very straightforward and I think quite widely used, so it's probably worth mentioning - perhaps in an addendum?

[sungam3r]

batch requests will be superseded by HTTP2

Maybe yes maybe no. In any case, whole years may pass until this comes. But batching is already working. And I must say that his support does not carry any burden.

[michaelstaib]

@benjie

be superseded by HTTP2 so I'm not sure how

Not really, we have implemented batching with the export directive where you can export results from the a preceding query as variables in the next.

And I think the PayPal guys were looking or discussing to use that for transactions. You can do with this nice mutation flows.

POST /graphql?batchOperations=[NewsFeed, StoryComments]

query NewsFeed {
  stories {
    id @export(as: "ids")
    actor
    message
  }
}

query StoryComments {
  stories(ids: $ids) {
    comments(first: 2) {
      actor
      message
    }
  }
}

We support two ways to batch ... operation batching like the above where you send in a document and specify the operations as a list in the url to specify which operation you want to batch...

And we also support the Apollo way which basically is a request batching. Batching for the sake to safe requests might not be necessary with HTTP/2 but combined with @export I see value in it for especially mutations.

[benjie]

@michaelstaib Sure; I was referring to straight-forward vanilla batching rather than with dependencies and chaining. Is there a full spec for the approach you raised? It seems you’d need a runtime on top of GraphQL to validate the types of the variables/exports before running any of the batch? That’s quite a but more complex than the simple array-based parallel batching that Apollo Server, PostGraphile and the like support.

[michaelstaib]

@benjie

Hot Chocolate and Sangria have implemented it this way. You actually do not need a lot extra. Directives in Hot Chocolate can be associated with field middleware, so the execution is quite simple. Validation wise you can also do static analysis to checkout if outputs are compatible with the inputs. We basically infer the variables.

Oleg and I both inferred the specification from a talk that lee gave :)

There are some discussions also with the GraphQL-Java guys about this on the spec repo. Also there are some discussions on this with Ivan, the PayPal guys and so on

I really should put that all together to a spec document. I think I will write a little blog summing all up and then we have something with all the information in one document.

https://hotchocolate.io/docs/batching

On our docs is the talk from @leebyron. They introduced this back then to basically solve something that you actually better solve with @defer. But, as I said, the superpower of @export and batching lies more in mutations in my opinion.

[sungam3r]

Just a note - I was talking about array-based batching approach.

[benjie]

These seem to be radically different topics that share some similarities.

With auto-batching the requests are captured and batched together as a purely transport-level optimisation, to reduce roundtrips. They introduce no new validation, GraphQL re-writing, or any other complexities. The operations typically run independently in parallel on the server, and the results are returned as an array in the same order to the client. They have trade-offs, and HTTP/2 (or using a websocket transport) solves the problem better.

With chained batches, the queries in the batch have to be prepared in advance so that they can be issued together, otherwise later operations may not validate as they rely on variables that are dynamically declared from previous operations' @export directives. They require a runtime to re-write the queries based on the @export directives, and must be ran serially (though there may be some parallelism depending on the variable dependencies) so the variables can be updated. The operations themselves are not valid GraphQL operations until the transforms have been applied, so many GraphQL validation/linting tools would reject them.

(By the way, I think the validation issues/rewrite requirements of these chained batches could be solved by changing the syntax for query StoryComments { to query StoryComments($ids: Int[] @import(from: "ids")) {.)

@sungam3r I suggest you rename this issue to "HTTP/1.1 roundtrip reduction via array-based auto-batching".

@michaelstaib I suggest you raise a new issue about the approach you mention, which I think might be better described as something like "chained operation batches".

[michaelstaib]

@benjie you are right and we should have multiple issues describing the different approaches.

[sungam3r]

@benjie I completely agree with your every word as if I wrote it myself.

[sungam3r]

The operations typically run independently in parallel on the server

Only one small remark - queries in the batch array may contain mutations, so it is unsafe to execute them in parallel. GraphQL-dotnet executes batched requests sequentially.

[benjie]

Interesting. I only use query auto-batching as a transport layer optimisation; so if, without batching, the application would have issued 4 operations at the same time over separate parallel HTTP requests (which would have ran independently and in parallel) then I treat it as acceptable to run those same operations independently and in parallel when received in a batch by the server. Making it such that mutations run in series does sound like a good idea though, even though it would make query batching even slower than it is already (taking it from as slow as the slowest operation, to as slow as the slowest chain of operations).

[michaelstaib]

@benjie

Just one question here, do you not write the result to the stream as they arrive.

We have optimized batching to write to the response stream as soon as possible. Our .NET client Strawberry Shake will then start reading the results from the stream as they arrive and make them immediately available.

So, we do not wait for the full array to be finished to be written to the stream. This means that the time to the first response is quite quick.

[michaelstaib]

@sungam3r

I think ordered execution for mutations is exactly what operation batching does. With that it is more explicit that you need an order.

The Apollo batching that is proposed here was about being able to merge multiple requests from a client into one in order to optimize transport. This is done by Apollo for instance under the hood.