Wrong custom header value

[fewensa]

When i try to call post request with custom header follow docs
https://docs.witnet.io/smart-contracts/witnet-web-oracle/make-a-post-request

const testPostSource = new Witnet.HttpPostSource(
    "https://httpbin.org/post",
    "This is the request body",
    {
        "Header-Name": "Header-Value"
    }
)
    .parseJsonMap()
    .getMap("headers")
    .getString("Header-Name")

got this response

 |      Execution trace:
 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/json\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646dd9b0-20d177407637005c73bab8d1\"\n  }, \n  \"json\": null, \n  \"origin\": \"180.109.166.147\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"
 │        [1] .parseJSONMap() -> Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}
 |      Result: Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Aggregation stage                              │
 │  ├────────────────────────────────────────────────┤
 │  │ Execution time: 0.22196 ms                     │
 │  │ Result is Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}│
 │  └────────────────────────────────────────────────┘
 │  
 │  ┌────────────────────────────────────────────────┐
 └──┤ Tally stage                                    │
    ├────────────────────────────────────────────────┤
    │ Execution time: 0.175298 ms                    │
    │ Result is Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/json\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646dd9b0-20d177407637005c73bab8d1"}}},"origin":{"RadonString":"180.109.166.147"},"url":{"RadonString":"https://httpbin.org/post"}}│

The reason is some where changed header value

{
	"args": {
		"RadonMap": {}
	},
	"data": {
		"RadonString": "This is the request body"
	},
	"files": {
		"RadonMap": {}
	},
	"form": {
		"RadonMap": {}
	},
	"headers": {
		"RadonMap": {
			"Accept": {
				"RadonString": "*/*"
			},
			"Accept-Encoding": {
				"RadonString": "deflate, gzip"
			},
			"Content-Length": {
				"RadonString": "24"
			},
			"Content-Type": {
				"RadonString": "[\"application/json\"]"
			},
			"Header-Name": {
				"RadonString": "[\"Header-Value\"]"
			},
			"Host": {
				"RadonString": "httpbin.org"
			},
			"User-Agent": {
				"RadonString": "[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Edg/92.0.902.78\"]"
			},
			"X-Amzn-Trace-Id": {
				"RadonString": "Root=1-646dd9b0-20d177407637005c73bab8d1"
			}
		}
	},
	"origin": {
		"RadonString": "180.109.166.147"
	},
	"url": {
		"RadonString": "https://httpbin.org/post"
	}
}

The head value changed to ["Header-Value"] and ["application/json"] not raw value.

[aesedepece]

Hi @fewensa thanks for reporting! I'm trying to reproduce this error and will keep you posted of any findings. Stay tuned!

[aesedepece]

For a start, I have updated the documentation so it mentions .parseJSONMap, not .parseJsonMap.

[aesedepece]

@fewensa I can see that header values are being changed to arrays containing the actual value as their only entry.

I believe this is done internally by the library we use for HTTP queries. I'll investigate some more and confirm where is this conversion happening.

I would also love to know more about your specific use case so that we can prototype the best way to deal with this.

[aesedepece]

@fewensa For what it's worth, you can simply add this at the end of the retrieval script to get rid of the extra brackets:

    .parseJSONArray()
    .getString(0)

Here's the full execution trace:

╔════════════════════════════════════════════╗
║ Witnet query local execution report        ║
╚╤═══════════════════════════════════════════╝
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Retrieval stage                                │
 │  ├────────────────────────────────────────────────┤
 │  │ Number of retrieved data sources: 1            │
 │  └┬───────────────────────────────────────────────┘
 │   │
 │   └─[ Source #0 (httpbin.org) ]
 |      Method: HTTP-POST
 |      Complete URL: https://httpbin.org/post
 |      Headers: 
 |        "Header-Name": "Header-Value"
 |      Body: This is the request body
 |      Number of executed operators: 5
 |      Execution time: 0.056541 ms
 |      Execution trace:
 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/octet-stream\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646e1a89-72de8699175814593f144d0d\"\n  }, \n  \"json\": null, \n  \"origin\": \"139.47.74.45\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"
 │        [1] .parseJSONMap() -> Map: {"args":{"RadonMap":{}},"data":{"RadonString":"This is the request body"},"files":{"RadonMap":{}},"form":{"RadonMap":{}},"headers":{"RadonMap":{"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/octet-stream\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646e1a89-72de8699175814593f144d0d"}}},"origin":{"RadonString":"139.47.74.45"},"url":{"RadonString":"https://httpbin.org/post"}}
 │        [2] .getMap(headers) -> Map: {"Accept":{"RadonString":"*/*"},"Accept-Encoding":{"RadonString":"deflate, gzip"},"Content-Length":{"RadonString":"24"},"Content-Type":{"RadonString":"[\"application/octet-stream\"]"},"Header-Name":{"RadonString":"[\"Header-Value\"]"},"Host":{"RadonString":"httpbin.org"},"User-Agent":{"RadonString":"[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\"]"},"X-Amzn-Trace-Id":{"RadonString":"Root=1-646e1a89-72de8699175814593f144d0d"}}
 │        [3] .getString(Header-Name) -> String: "[\"Header-Value\"]"
 │        [4] .parseJSONArray() -> Array: [{"RadonString":"Header-Value"}]
 │        [5] .getString(0) -> String: "Header-Value"
 |      Result: String: "Header-Value"
 │
 │  ┌────────────────────────────────────────────────┐
 ├──┤ Aggregation stage                              │
 │  ├────────────────────────────────────────────────┤
 │  │ Execution time: 0.00653 ms                     │
 │  │ Result is String: "Header-Value"               │
 │  └────────────────────────────────────────────────┘
 │  
 │  ┌────────────────────────────────────────────────┐
 └──┤ Tally stage                                    │
    ├────────────────────────────────────────────────┤
    │ Execution time: 0.00204 ms                     │
    │ Result is String: "Header-Value"               │
    └────────────────────────────────────────────────┘

[fewensa]

Hi @aesedepece Thanks for your help, I think now don't care how to parse response, because this has no value, this data is the request data structure returned by httpbin.org.

 │        [0] HTTP-POST -> String: "{\n  \"args\": {}, \n  \"data\": \"This is the request body\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"24\", \n    \"Content-Type\": \"[\\\"application/octet-stream\\\"]\", \n    \"Header-Name\": \"[\\\"Header-Value\\\"]\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"[\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15\\\"]\", \n    \"X-Amzn-Trace-Id\": \"Root=1-646e1a89-72de8699175814593f144d0d\"\n  }, \n  \"json\": null, \n  \"origin\": \"139.47.74.45\", \n  \"url\": \"https://httpbin.org/post\"\n}\n"

\"Content-Type\": \"[\\\"application/octet-stream\\\"]\"

As long as you still see an array here, it means that the request is wrong.

Wrong header

$ curl -vvv -X POST -H 'Content-Type: ["application/json"]' https://pangolin-rpc.darwinia.network -d '{"jsonrpc":"2.0","id":"1","method":"chain_getFinalizedHead"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 198.18.2.221:443...
* Connected to pangolin-rpc.darwinia.network (198.18.2.221) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> POST / HTTP/1.1
> Host: pangolin-rpc.darwinia.network
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Type: ["application/json"]
> Content-Length: 60
>
< HTTP/1.1 415 Unsupported Media Type
< Date: Wed, 24 May 2023 09:36:20 GMT
< Content-Type: text/plain
< Content-Length: 81
< Connection: keep-alive
< access-control-allow-origin: *
< vary: origin
< vary: access-control-request-method
< vary: access-control-request-headers
< access-control-expose-headers: *
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 7cc48dbc8acd0499-HKG
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
Supplied content type is not allowed. Content-Type: application/json is required
* Connection #0 to host pangolin-rpc.darwinia.network left intact

Right header

$ curl -vvv -X POST -H 'Content-Type: application/json' https://pangolin-rpc.darwinia.network -d '{"jsonrpc":"2.0","id":"1","method":"chain_getFinalizedHead"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 198.18.2.221:443...
* Connected to pangolin-rpc.darwinia.network (198.18.2.221) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> POST / HTTP/1.1
> Host: pangolin-rpc.darwinia.network
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 60
>
< HTTP/1.1 200 OK
< Date: Wed, 24 May 2023 09:36:52 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 104
< Connection: keep-alive
< access-control-allow-origin: *
< vary: origin
< vary: access-control-request-method
< vary: access-control-request-headers
< access-control-expose-headers: *
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 7cc48e82cbc50500-HKG
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{"jsonrpc":"2.0","result":"0x18936a14745b23b1aa9205d486db58181715f8b84cd57c2e4dafd49f133c8eba","id":"1"}

I believe this is done internally by the library we use for HTTP queries. I'll investigate some more and confirm where is this conversion happening.

I think so, I tried to investigate the code of witnet-requests-js. I didn't find where the http request was called.

witnet-requests-js/src/lib/radon/stages.js

Line 26 in 6a03a66

this.headers = headers && Object.entries(headers).map(([key, value]) => ({ left: key, right: value }));

[fewensa]

I think there changed header value to array. then send data direct use sede_json::to_string, send raw json string to server.

https://github.com/witnet/witnet-rust/blob/7e00113eae71ced5119474f7ebc4e74d23b5b34c/rad/src/lib.rs#L369-L376

https://docs.rs/http-types/2.12.0/http_types/headers/trait.ToHeaderValues.html#tymethod.to_header_values

https://docs.rs/http-types/2.12.0/http_types/headers/struct.HeaderValues.html

[fewensa]

https://cloud.google.com/apigee/docs/api-platform/antipatterns/multi-value-http-headers?hl=en