Parsing HTTP Request/Response Headers

[bbgdzxng1]

I wish to do some analysis on some HTTP Response Headers, specifically looking at access-control headers. It would make sense to try to put all headers into a navigable JSON, with the ultimate aim of navigating and validating the HTTP response headers with jq. I'm sure that security auditors have their own custom scripts for parsing headers.

Here's a typical curl --head request, showing the HTTP headers.

$ curl --silent --header "origin: https://origin.example.com" --request GET "https://cors-test.appspot.com/test" --head | grep -e ":"
cache-control: no-cache
access-control-allow-origin: https://origin.example.com
access-control-max-age: 0
access-control-allow-credentials: true
set-cookie: test=test
expires: Fri, 01 Jan 1990 00:00:00 GMT
content-type: application/json
x-cloud-trace-context: a7b21a2d8560b32c87c09bf8019f2813
date: Sat, 02 Apr 2022 04:13:03 GMT
server: Google Frontend
content-length: 15
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

I have got as far as using tr (or $IFS) to replace colons with commas and using the generic jc --csv parser, however that approach is too naive for HTTP headers such as date: Sat, 02 Apr 2022 03:09:19 GMT or alt-svc, and also CSV also requires column headers.

A more expansive, generic approach would be parsing both curl --verbose Request/Response headers, which are indicated by a > or <.

$ curl --silent --header "origin: https://origin.example.com" --request GET "https://cors-test.appspot.com/test" -o /dev/null --verbose 2>&1 | grep -e ">" -e "<"
> GET /test HTTP/2
> Host: cors-test.appspot.com
> User-Agent: curl/7.64.1
> Accept: */*
> origin: https://origin.example.com
> 
< HTTP/2 200 
< cache-control: no-cache
< access-control-allow-origin: https://origin.example.com
< access-control-max-age: 0
< access-control-allow-credentials: true
< set-cookie: test=test
< expires: Fri, 01 Jan 1990 00:00:00 GMT
< content-type: application/json
< x-cloud-trace-context: 22166a19657ba2b7eacaf5fe4a70c229
< date: Sat, 02 Apr 2022 04:12:20 GMT
< server: Google Frontend
< content-length: 15
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< 

Obviously, the ultimate solution would be for cURL to present HTTP request/response headers in json, but in the interim, this feels like a job for the mighty jc. I was hoping to pre-parse prior to sending to jc, because I don't have any python-skills to write a custom parser in python.

Does anyone have any advice as to pre-parse cURL --head or --verbose into a format that jc can eat in order to access request/response HTTP headers in structured data?

Thanks!

[kellyjonbrazil]

Hi there!

Yeah, this looks pretty cool. I did a couple quick tests and found that the Key/Value parser (--kv) seems to do a quick-and-dirty job of parsing this output:

$ echo 'cache-control: no-cache
> access-control-allow-origin: https://origin.example.com
> access-control-max-age: 0
> access-control-allow-credentials: true
> set-cookie: test=test
> expires: Fri, 01 Jan 1990 00:00:00 GMT
> content-type: application/json
> x-cloud-trace-context: a7b21a2d8560b32c87c09bf8019f2813
> date: Sat, 02 Apr 2022 04:13:03 GMT
> server: Google Frontend
> content-length: 15
> alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"' | jc --kv -p
{
  "cache-control": "no-cache",
  "access-control-allow-origin": "https://origin.example.com",
  "access-control-max-age": "0",
  "access-control-allow-credentials": "true",
  "set-cookie": "test=test",
  "expires": "Fri, 01 Jan 1990 00:00:00 GMT",
  "content-type": "application/json",
  "x-cloud-trace-context": "a7b21a2d8560b32c87c09bf8019f2813",
  "date": "Sat, 02 Apr 2022 04:13:03 GMT",
  "server": "Google Frontend",
  "content-length": "15",
  "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
}

and

$ echo '> GET /test HTTP/2
> > Host: cors-test.appspot.com
> > User-Agent: curl/7.64.1
> > Accept: */*
> > origin: https://origin.example.com
> > 
> < HTTP/2 200 
> < cache-control: no-cache
> < access-control-allow-origin: https://origin.example.com
> < access-control-max-age: 0
> < access-control-allow-credentials: true
> < set-cookie: test=test
> < expires: Fri, 01 Jan 1990 00:00:00 GMT
> < content-type: application/json
> < x-cloud-trace-context: 22166a19657ba2b7eacaf5fe4a70c229
> < date: Sat, 02 Apr 2022 04:12:20 GMT
> < server: Google Frontend
> < content-length: 15
> < alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
> < ' | jc --kv -p
{
  "> get /test http/2": "",
  "> host": "cors-test.appspot.com",
  "> user-agent": "curl/7.64.1",
  "> accept": "*/*",
  "> origin": "https://origin.example.com",
  ">": "",
  "< http/2 200": "",
  "< cache-control": "no-cache",
  "< access-control-allow-origin": "https://origin.example.com",
  "< access-control-max-age": "0",
  "< access-control-allow-credentials": "true",
  "< set-cookie": "test=test",
  "< expires": "Fri, 01 Jan 1990 00:00:00 GMT",
  "< content-type": "application/json",
  "< x-cloud-trace-context": "22166a19657ba2b7eacaf5fe4a70c229",
  "< date": "Sat, 02 Apr 2022 04:12:20 GMT",
  "< server": "Google Frontend",
  "< content-length": "15",
  "< alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
  "<": ""
}

Though a curl parser could be created to do a cleaner job. I thought curl introduced a JSON output option in one of the latest releases?

[kellyjonbrazil]

I did some more testing and ended up needing to run the curl output through uniq since there were duplicate headers that seemed to gum up the Key/Value parser.:

$ curl --silent --header "origin: https://origin.example.com"\
            --request GET "https://cors-test.appspot.com/test"\
            --head | grep -e ":" | uniq | jc --kv -p
{
  "cache-control": "no-cache",
  "access-control-allow-origin": "https://origin.example.com",
  "access-control-max-age": "0",
  "access-control-allow-credentials": "true",
  "set-cookie": "test=test",
  "expires": "Fri, 01 Jan 1990 00:00:00 GMT",
  "content-type": "application/json",
  "x-cloud-trace-context": "1cea9464d68b0f7481b21ea8f67addb5",
  "date": "Sat, 02 Apr 2022 18:16:35 GMT",
  "server": "Google Frontend",
  "content-length": "15",
  "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
}

[kellyjonbrazil]

The next release of jc (v1.18.7) will include a fix to the Key/Value parser such that duplicate keys will be allowed, yet only the last value is used. #219

[kellyjonbrazil]

I'm currently working on an HTTP Header parser and a curl --head parser. These parsers handle curl's verbose output (so you can also grab request headers) and properly handle headers that can be defined multiple times and include multiple values delimited by a comma.

This is currently in the dev branch and will be released in the next version of jc:

https://github.com/kellyjonbrazil/jc/blob/dev/jc/parsers/http_headers.py

https://github.com/kellyjonbrazil/jc/blob/dev/jc/parsers/curl_head.py

[kellyjonbrazil]

The curl-head and http-headers parsers were released in jc v1.25.0