Skip to content

Updated conformance tests to validate the connect protocol specification content-type updates - #1191

Merged
timostamm merged 17 commits into
connectrpc:mainfrom
jonbodner-buf:jbodner/content-type-tests
Nov 11, 2026
Merged

Updated conformance tests to validate the connect protocol specification content-type updates#1191
timostamm merged 17 commits into
connectrpc:mainfrom
jonbodner-buf:jbodner/content-type-tests

Conversation

@jonbodner-buf

@jonbodner-buf jonbodner-buf commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

connectrpc/connectrpc.com#447 updates the connect protocol specification to clarify how content-type parameters should be handled. This PR includes conformance tests to validate that implementations consistently follow those rules.

The reference implementation (connect-go) does not currently meet the updated specification, so there are some known failing conformance tests. A future update to connect-go will resolve these.

jonbodner-buf and others added 15 commits October 16, 2026 17:05
@jonbodner-buf
update to connect-go v2
f95f680 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
updates in response to feedback
02b80cc 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@timostamm
Merge branch 'main' into jbodner/connect-go-v2
5be23a7
@jonbodner-buf
properly handle grpcWeb when there's an immediate error. respect trai…
f19fe95 
…lers-only.

Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
Merge branch 'jbodner/connect-go-v2' of github.com:jonbodner-buf/conf…
daf4a47 
…ormance into jbodner/connect-go-v2
@jonbodner-buf
update to connect-go v2 2.0.2 for SendHeader fix. Add test to validat…
8a8e1e8 
…e headers are sent immediately now.

Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
in referenceclient, make sure that we don't try to modify a nil resul…
cd9ea78 
…t in a defer

Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
add new conformance tests to validate content-type handling. update t…
5643c89 
…he known failing until connect-go is updated as well.

Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
add additional conformance tests to validate that proto messages igno…
0631a2a 
…re charset

Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
add test for unknown code on 415
2c8e29c 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
do not apply connect content type conformance rules to grpc
15ec252 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
merge main
1d05ea1 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
remove grpc content-type tests
5246aa3 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
remove grpc content-type tests from known failing
99c3eff 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@jonbodner-buf
update known failing tests until connect-go is updated
dfd033a 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@github-actions

github-actions Bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Implementation conformance

connect-go, the reference implementation, has newly failing tests. They predict failures in its own CI.

Implementation Newly failing test cases Newly passing test cases Known-failing list Notes
connect-dart 91b0cb4 4 in 2 of 2 targets 0 no list
connect-es c6b7be8 26 in 6 of 6 targets 0 add 26
connect-go 297fdd6 (reference) 23 in 2 of 2 targets 0 no list
connect-kotlin a70a2d7 13 in 1 of 1 targets 0 add 13
connect-py d72bde3 13 in 2 of 2 targets 0 no list
connect-rust bd2ab67 8 in 2 of 2 targets 0 no list
connect-swift 8d52280 9 in 1 of 1 targets 0 no list

Changed suites: HTTP to RPC Code Mapping, Connect JSON Content-Type Parameter Responses, Connect Proto Content-Type Parameter Responses, Connect JSON Content-Type Parameter Requests, Connect Proto Content-Type Parameter Requests.

Each implementation ran the suites with and without this pull request's changes, without known-failing lists, at the commit shown. Test cases whose results differed, or that couldn't run, ran once more with and without the changes (only with them for added suites), and those whose differences didn't repeat are listed as possibly flaky instead of newly failing or passing. The new failures and passes were then checked against the known-failing list each target passes to the runner. Results for dcfa50786ceb from this workflow run.

Newly failing test cases

Test cases that aren't in the known-failing list need an entry there, or a fix.

connect-dart
Target Test case Permutations In known-failing list
io Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-unsupported-charset 3 no
io Connect JSON Content-Type Parameter Responses/server-stream/unsupported-charset 3 no
io Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 3 no
io Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 3 no
web Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-unsupported-charset 3 no
web Connect JSON Content-Type Parameter Responses/server-stream/unsupported-charset 3 no
web Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 3 no
web Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 3 no
connect-es
Target Test case Permutations In known-failing list
node client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-quoted 4 no
node client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-with-unknown-parameter 4 no
node client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter 4 no
node client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-charset-utf-8 4 no
node client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-unsupported-charset 4 no
node client Connect JSON Content-Type Parameter Responses/server-stream/unsupported-charset 4 no
node client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-quoted 4 no
node client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-with-unknown-parameter 4 no
node client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter 4 no
node client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-charset-utf-8 4 no
node client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 4 no
node client Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 4 no
node client Connect Proto Content-Type Parameter Responses/server-stream/unknown-parameter 4 no
node client Connect Proto Content-Type Parameter Responses/unary/unknown-parameter 4 no
node server Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-quoted 4 no
node server Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-with-unknown-parameter 4 no
node server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter 4 no
node server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-charset-utf-8 4 no
node server Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-quoted 4 no
node server Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-with-unknown-parameter 4 no
node server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter 4 no
node server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-charset-utf-8 4 no
node server Connect Proto Content-Type Parameter Requests/server-stream/charset-ignored 4 no
node server Connect Proto Content-Type Parameter Requests/server-stream/unknown-parameter 4 no
node server Connect Proto Content-Type Parameter Requests/unary/charset-ignored 4 no
node server Connect Proto Content-Type Parameter Requests/unary/unknown-parameter 4 no
fastify Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-quoted 4 no
fastify Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-with-unknown-parameter 4 no
fastify Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter 4 no
fastify Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-charset-utf-8 4 no
fastify Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-quoted 4 no
fastify Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-with-unknown-parameter 4 no
fastify Connect JSON Content-Type Parameter Requests/unary/unknown-parameter 4 no
fastify Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-charset-utf-8 4 no
fastify Connect Proto Content-Type Parameter Requests/server-stream/charset-ignored 4 no
fastify Connect Proto Content-Type Parameter Requests/server-stream/unknown-parameter 4 no
fastify Connect Proto Content-Type Parameter Requests/unary/charset-ignored 4 no
fastify Connect Proto Content-Type Parameter Requests/unary/unknown-parameter 4 no
express Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-quoted 2 no
express Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-with-unknown-parameter 2 no
express Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter 2 no
express Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-charset-utf-8 2 no
express Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-quoted 2 no
express Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-with-unknown-parameter 2 no
express Connect JSON Content-Type Parameter Requests/unary/unknown-parameter 2 no
express Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-charset-utf-8 2 no
express Connect Proto Content-Type Parameter Requests/server-stream/charset-ignored 2 no
express Connect Proto Content-Type Parameter Requests/server-stream/unknown-parameter 2 no
express Connect Proto Content-Type Parameter Requests/unary/charset-ignored 2 no
express Connect Proto Content-Type Parameter Requests/unary/unknown-parameter 2 no

Only 50 of 78 are shown. The job summaries list all of them.

connect-go
Target Test case Permutations In known-failing list
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-quoted 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-uppercase 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-with-unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-charset-utf-8 5 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-quoted 5 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-uppercase 5 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-with-unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-charset-utf-8 5 no
client Connect Proto Content-Type Parameter Responses/server-stream/unknown-parameter 5 no
client Connect Proto Content-Type Parameter Responses/unary/unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/server-stream/charset-utf-8-with-unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-charset-utf-8 5 no
server Connect JSON Content-Type Parameter Requests/unary/charset-utf-8-with-unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-charset-utf-8 5 no
server Connect Proto Content-Type Parameter Requests/server-stream/charset-ignored 5 no
server Connect Proto Content-Type Parameter Requests/server-stream/unknown-parameter 5 no
server Connect Proto Content-Type Parameter Requests/unary/charset-ignored 5 no
server Connect Proto Content-Type Parameter Requests/unary/unknown-parameter 5 no
connect-kotlin
Target Test case Permutations In known-failing list
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-quoted 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-uppercase 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-with-unknown-parameter 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-charset-utf-8 4 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-quoted 4 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-uppercase 4 no
client Connect JSON Content-Type Parameter Responses/unary/charset-utf-8-with-unknown-parameter 4 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter 4 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-charset-utf-8 4 no
client Connect Proto Content-Type Parameter Responses/server-stream/unknown-parameter 4 no
client Connect Proto Content-Type Parameter Responses/unary/unknown-parameter 4 no
connect-py
Target Test case Permutations In known-failing list
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-quoted 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-uppercase 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-with-unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter 5 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-charset-utf-8 5 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 5 no
client Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 5 no
client Connect Proto Content-Type Parameter Responses/server-stream/unknown-parameter 5 no
server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-unsupported-charset 5 no
server Connect JSON Content-Type Parameter Requests/server-stream/unsupported-charset 5 no
server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-unsupported-charset 5 no
server Connect JSON Content-Type Parameter Requests/unary/unsupported-charset 5 no
connect-rust
Target Test case Permutations In known-failing list
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-unsupported-charset 4 no
client Connect JSON Content-Type Parameter Responses/server-stream/unsupported-charset 4 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 4 no
client Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 4 no
server Connect JSON Content-Type Parameter Requests/server-stream/unknown-parameter-then-unsupported-charset 2 no
server Connect JSON Content-Type Parameter Requests/server-stream/unsupported-charset 2 no
server Connect JSON Content-Type Parameter Requests/unary/unknown-parameter-then-unsupported-charset 2 no
server Connect JSON Content-Type Parameter Requests/unary/unsupported-charset 2 no
connect-swift
Target Test case Permutations In known-failing list
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8 2 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-quoted 2 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-uppercase 2 no
client Connect JSON Content-Type Parameter Responses/server-stream/charset-utf-8-with-unknown-parameter 2 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter 2 no
client Connect JSON Content-Type Parameter Responses/server-stream/unknown-parameter-then-charset-utf-8 2 no
client Connect JSON Content-Type Parameter Responses/unary/unknown-parameter-then-unsupported-charset 2 no
client Connect JSON Content-Type Parameter Responses/unary/unsupported-charset 2 no
client Connect Proto Content-Type Parameter Responses/server-stream/unknown-parameter 2 no

@jonbodner-buf
add tests for cases where charset comes second
7a43124 
Signed-off-by: Jon Bodner <jbodner@buf.build>
@anuraaga anuraaga mentioned this pull request Nov 11, 2026
Merged
@jonbodner-buf jonbodner-buf mentioned this pull request Nov 11, 2026
Merged
@anuraaga anuraaga mentioned this pull request Nov 11, 2026
Merged
@anuraaga

anuraaga commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

/rerun connect-py connectrpc/connect-py#464

@github-actions

github-actions Bot commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Implementation conformance

Rerun of connect-py at connectrpc/connect-py#464, requested in this comment.

No tests newly fail.

Implementation Newly failing test cases Newly passing test cases Known-failing list Notes
connect-py 2cba064 0 0 no list

Changed suites: HTTP to RPC Code Mapping, Connect JSON Content-Type Parameter Responses, Connect Proto Content-Type Parameter Responses, Connect JSON Content-Type Parameter Requests, Connect Proto Content-Type Parameter Requests.

Each implementation ran the suites with and without this pull request's changes, without known-failing lists, at the commit shown. Test cases whose results differed, or that couldn't run, ran once more with and without the changes (only with them for added suites), and those whose differences didn't repeat are listed as possibly flaky instead of newly failing or passing. The new failures and passes were then checked against the known-failing list each target passes to the runner. Results for 7a431244cc31 from this workflow run.

@timostamm
timostamm merged commit c91ac8b into connectrpc:main Nov 11, 2026
24 checks passed
emcfarlane pushed a commit to connectrpc/connect-go that referenced this pull request Nov 11, 2026
@jonbodner-buf
Meet the updated content-type and timeout spec changes (#1066)
c664348 
There are spec updates for content-type and timeout:

connectrpc/connectrpc.com#447
connectrpc/connectrpc.com#429

There are also conformance test updates to validate the spec updates:

connectrpc/conformance#1193
connectrpc/conformance#1191

This PR updates connect-go to implement the spec updates and pass the
conformance tests.

---------

Signed-off-by: Jon Bodner <jbodner@buf.build>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timostamm timostamm timostamm approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jonbodner-buf @anuraaga @timostamm