Directly encoding a ujson object silently activates chunked upload

[fhackett]

When directly sending a ujson object (relying on the geny based encoding feature), the requests library defaults to adding what I think are HTTP/2 streaming markers. This is fine for an endpoint that supports HTTP/2, because the standard allows the client to switch to the HTTP/2 encoding without checking that the server actually supports it. When communicating with a genuine HTTP/1.1 only server, this causes a parsing error when receiving the payload, because the additional data is not valid JSON (which the endpoint would expect to parse, not knowing its actual significance).

Here is a simple example:

//> using dependency "com.lihaoyi::requests:0.9.0"
//> using dependency "com.lihaoyi::upickle:4.2.1"

requests.post(
  "http://localhost:3000/",
  data = ujson.Obj(),
)

Most endpoints actually support HTTP/2 and this will often just work, so we need to look at the actual request using something like npx http-echo-server. Using that utility, we see this:

[server] event: connection (socket#2)
[socket#2] event: resume
[socket#2] event: data
--> POST / HTTP/1.1
--> Connection: Upgrade, HTTP2-Settings
--> Host: localhost:3000
--> HTTP2-Settings: AAEAAEAAAAIAAAAAAAMAAAAAAAQBAAAAAAUAAEAAAAYABgAA
--> Transfer-encoding: chunked
--> Upgrade: h2c
--> Accept: */*
--> Accept-Encoding: gzip, deflate
--> Content-Type: application/json
--> User-Agent: requests-scala
--> 
--> 
[socket#2] event: data
--> 2
--> {}
--> 
[socket#2] event: data
--> 0
--> 
--> 
[socket#2] event: prefinish
[socket#2] event: finish
[socket#2] event: readable
[socket#2] event: end
[socket#2] event: close

If the server is HTTP/2 compliant, this is a supported behavior and it will work (the app will see {}). However, for an HTTP/1.1 endpoint, the application layer will really see something like 2\n{}\n\n0, rather than {}.

To show that this is specifically how the JSON gets encoded, and not some more fundamental bug in the underlying Java library, I can add .toString, manually specify the content type, and the request changes significantly.

//> using dependency "com.lihaoyi::requests:0.9.0"
//> using dependency "com.lihaoyi::upickle:4.2.1"

requests.post(
  "http://localhost:3000/",
  headers = Map(
    "Content-Type" -> "application/json",
  ),
  data = ujson.Obj().toString,
)

Now, we can still see it try to negotiate HTTP/2, but the extra stream markers are gone, and a pure HTTP/1.1 server will accept the request, ignore the negotiation it doesn't understand, and all is fine.

[socket#3] event: resume
[socket#3] event: data
--> POST / HTTP/1.1
--> Connection: Upgrade, HTTP2-Settings
--> Content-Length: 2
--> Host: localhost:3000
--> HTTP2-Settings: AAEAAEAAAAIAAAAAAAMAAAAAAAQBAAAAAAUAAEAAAAYABgAA
--> Upgrade: h2c
--> Accept: */*
--> Accept-Encoding: gzip, deflate
--> Content-Type: application/json
--> User-Agent: requests-scala
--> 
--> 
[socket#3] event: data
--> {}
[socket#3] event: prefinish
[socket#3] event: finish
[socket#3] event: readable
[socket#3] event: end
[socket#3] event: close

This is quite a confusing issue, which manifests in practice as an oddity like "why does literally the same request work with curl and not scala-requests?". It would be nice to see a fix, but at least I hope this issue documents the problem and my workaround: just use .toString, which is fine for my simple use case sending a few bytes of JSON to trigger an event.

[matsilva]

Ran into similar problem, dug in and I think this is what is going on.

Problem

The Java 11 HttpClient defaults to attempting HTTP/2 upgrades by sending these headers:

Connection: Upgrade, HTTP2-Settings
HTTP2-Settings: AAEAAEAAAAIAAAAAAAAAAAAAAAAQBAAAAAAUAAEAAAAYABgAA
Upgrade: h2c

Many local services (LM Studio, Ollama, local APIs) don't properly handle HTTP/2 upgrade requests, causing connections to hang until timeout.

At present the requests-scala lib does not provide the user a way to specify a forced http version.

Proposed Solution

Add the ability to specify the http version in Requester.scala apis.

 def apply(
//... other config
httpVersion = HttpClient.Version.HTTP_1_1 // will force HTTP/1.1, dont actually default to http 1, just an example
//... other config
)

then in the http client builder....

val httpClient: HttpClient =
  HttpClient
    .newBuilder()
    .version(httpVersion)
    .followRedirects(HttpClient.Redirect.NEVER)
    // ... rest of configuration

If i get time soon (unlikely), i will take a crack at a pr

[lihaoyi]

@matsilva do send a PR if you can!

[fhackett]

I figured I could at least draft a failing test case for this (thanks for the ping btw), but I actually found something more confusing as a result.
Just to see what happened, I hardcoded @matsilva 's suggestion (just the builder part) in a local checkout, and I got this when echoing the resulting HTTP protocol:

[server] event: listening (port: 3000)
[server] event: connection (socket#1)
[socket#1] event: resume
[socket#1] event: data
--> POST /echo HTTP/1.1
--> Host: localhost:3000
--> Transfer-encoding: chunked
--> Accept: */*
--> Accept-Encoding: gzip, deflate
--> Content-Type: application/json
--> User-Agent: requests-scala
--> 
--> 
[socket#1] event: data
--> 2
--> {}
--> 0
--> 
--> 
[socket#1] event: prefinish
[socket#1] event: finish
[socket#1] event: readable
[socket#1] event: end
[socket#1] event: close

Notice that the mysterious 2 is still there.

What's even weirder is that, when I tried to make a test with ServerUtils.usingEchoServer (it uses com.sun.net.httpserver internally, which supports only HTTP/1), the test passes. As in, the echo I get back is just {}. Color me confused.

[fhackett]

... I just figured out what was going on. For some reason, I was talking to an endpoint that didn't support HTTP/1.1 chunking (apparently it happens, news to me). It's a different feature than HTTP/2 multiplexing, and it's a coincidence that all the docs I found suggested I was seeing the HTTP/2 behavior. My bad.

Now I know to look for it, the 2 above is a chunk header indicating the 2 byte long JSON data. It also says Transfer-encoding: chunked right there.

I would say "mystery solved", but actually there is still something that doesn't make sense. I found the config option chunkedUpload, and not only should it default to false, but when I set it to false manually, I still see a chunked upload attempt (behavior is unchanged from above).

I looked around the source code, and, unless I am missing something, I only see chunkedUpload being passed around, not actually used for anything. Looking more closely into HttpClient behavior, I think the underlying library decides which upload type to use purely based on knowing content length. ujson.Value doesn't advertise a known length, so it gets chunked in all cases. Making it a string brings back the known length, so no chunking.

If we want this option to work again, it seems we'd need possible internal buffering so HttpClient gets its content length (if chunkedUpload = false). I found other libraries that do this, so it doesn't seem unthinkable. On the other hand, chunkedUpload's value defaults to false but the behavior defaults to true regardless of the setting, so there's user expectations to think about.

Any thoughts? I also can't commit to a PR soon, but thanks to the new activity I was inspired to understand the issue better.

Forcing HTTP/1.1 also sounds useful along the theme of working with badly behaved servers, though I am becoming more sure we are talking about 2 related issues, not the same issue.

Perhaps the real challenge here is reliably testing all this, which might require deliberately sourcing one or more non compliant HTTP servers ... or at least one that echoes requests in detail.