[rfc6265bis] Cookie parser - UTF-8 chars

[bakulf]

Following the RFC6265bis section 4.1.1 a cookie-value can contain only ASCII chars:

   cookie-name       = token
   cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
   cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                         ; US-ASCII characters excluding CTLs,
                         ; whitespace DQUOTE, comma, semicolon,
                         ; and backslash

Modern browsers do allow UTF-8 chars too and we have web-platform-tests to enforce this behavior.
I would suggest extending the cookie-octet with the range %x80-FF.

[annevk]

Thanks for raising this, I think this is basically whatwg/html#804 again, for which we had #159, but that didn't cover everything that was identified there.

cc @inikulin

[inikulin]

I think this is basically whatwg/html#804 again, for which we had #159, but that didn't cover everything that was identified there.

Correct. Though, at the time of testing only FF and Chrome supported UTF-8 properly, so it wasn't considered as a de-facto standard.

[mikewest]

Non-ASCII cookies are certainly produced by servers out there in the wild, and accepted by several user agents (though not, apparently, Safari: https://wpt.fyi/results/cookies/http-state/charset-tests.html?label=experimental&label=master&aligned). I'm comfortable extending based on those test results.

[mikewest]

(That said, I don't think it's as simple as extending cookie-octet. We'll need to define the encoding, serialization, etc. in a little more detail if we're going to move outside of ASCII.)

[annevk]

Yeah, at the lowest level you're looking at two parsers, one that takes bytes and one that takes code points (potentially scalar values if we can make document.cookie a USVString). Which one feeds into which and how requires investigation.

[mnot]

Just a random comment - I would buy both of you some drinks if you could figure out a way to express the [Set-]Cookie header data model in terms of structured headers fields, and the define parsing and serialisation algorithms to get there...

[annevk]

It seems efaa468 (marked editorial) changed this to include 0x80-0xFF and that matches Chrome in that it does bytes-in-bytes-out. Firefox tries to decode to UTF-8 (so 0xFF becomes U+FFFD and three bytes in the Cookie header rather than staying 0xFF). Safari seems to truncate at the first non-ASCII byte which strikes me as pretty bad behavior?

The tests mentioned above should account for invalid UTF-8 as it seems they currently do not and therefore do not capture this difference between Chrome and Firefox. I suspect we should align Firefox with Chrome here unless there is interest to do the opposite?

(Firefox has seen at least one compatibility problem from not having the Chrome behavior here: https://bugzilla.mozilla.org/show_bug.cgi?id=1664702.)

[arichiv]

On it, more soon