Don't disable http tee after first header, so that headers of re-directs can be captured and we don't panic

[phillip-stephens]

Description

From what I can tell, the intention with this change originally was to tee the http response to efficiently get the raw headers. To save on performance, the tee was disabled after the header, since in theory there's no need to tee afterwards on the body.

The issue arrises when we re-direct. We attempt to read the tee'd buffer after being re-directed and the tee being disabled and which is now out-of-sync with the underlying buffer. This causes an attempt to access a slice out-of-bounds. Since we'd want to get the headers of re-directs as well, I think a simple fix of just leaving the tee enabled is sufficient.

How to Test

I manually checked the raw_header and ensured that once you base64 --decode, both the final response and re-direct raw_headers are correct when compared to Wireshark.

master

178.16.143.250 is an http site that redirects you. It panics with --raw-headers.

echo "178.16.143.250" | ./zgrab2 http --raw-headers --max-redirects=1 --with-body-size | jq 

$ echo "178.16.143.250" | ./zgrab2 http --raw-headers --max-redirects=1 --with-body-size | jq

INFO[0000] started grab at 2025-03-11T11:02:15-07:00
panic: runtime error: slice bounds out of range [:-737]

goroutine 1058 [running]:
github.com/zmap/zgrab2/lib/http.(*TeeConn).Bytes(...)
	/Users/phillip/zgrab2/lib/http/transport.go:1166
github.com/zmap/zgrab2/lib/http.readResponse(0x14000524000, 0x140001f8400)
	/Users/phillip/zgrab2/lib/http/response.go:226 +0x6b0
github.com/zmap/zgrab2/lib/http.ReadResponseTee(...)
	/Users/phillip/zgrab2/lib/http/response.go:169
github.com/zmap/zgrab2/lib/http.(*persistConn).readResponse(0x14000000240, {0x140001f8400, 0x140000a28c0, 0x1, 0x0, 0x140000a2850}, 0x0)
	/Users/phillip/zgrab2/lib/http/transport.go:1707 +0x7c
github.com/zmap/zgrab2/lib/http.(*persistConn).readLoop(0x14000000240)
	/Users/phillip/zgrab2/lib/http/transport.go:1549 +0x304
created by github.com/zmap/zgrab2/lib/http.(*Transport).dialConn in goroutine 1043
	/Users/phillip/zgrab2/lib/http/transport.go:1138 +0x1064

Branch

$ echo "178.16.143.250" | ./zgrab2 http --raw-headers --max-redirects=1 --with-body-size

INFO[0000] started grab at 2025-03-11T11:03:29-07:00
{"ip":"178.16.143.250","data":{"http":{"status":"success","protocol":"http","result":{"response":{"status_line":"200 OK","status_code":200,"protocol":{"name":"HTTP/1.1","major":1,"minor":1},"headers":{"accept_ranges":["bytes"],"content_type":["text/html"],"date":["Tue, 11 Mar 2025 18:03:29 GMT"],"etag":["\"dbf-61e34e1e8f5cd-gzip\""],"last_modified":["Sat, 27 Jul 2024 06:38:57 GMT"],"server":["Apache/2.4.58 (Ubuntu)"],"vary":["Accept-Encoding"]},"headers_raw":"SFRUUC8xLjEgMjAwIE9LDQpEYXRlOiBUdWUsIDExIE1hciAyMDI1IDE4OjAzOjI5IEdNVA0KU2VydmVyOiBBcGFjaGUvMi40LjU4IChVYnVudHUpDQpMYXN0LU1vZGlmaWVkOiBTYXQsIDI3IEp1bCAyMDI0IDA2OjM4OjU3IEdNVA0KRVRhZzogImRiZi02MWUzNGUxZThmNWNkLWd6aXAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KVmFyeTogQWNjZXB0LUVuY29kaW5nDQpDb250ZW50LUVuY29kaW5nOiBnemlwDQpDb250ZW50LUxlbmd0aDogMTA4Nw0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCg0K","body":"\u003c!DOCTYPE html\u003e\r\n\u003chtml lang=\"ko\"\u003e\r\n\u003chead\u003e\r\n    \u003cmeta charset=\"UTF-8\"\u003e\r\n    \u003cmeta content=\"width=device-width, initial-scale=1\" name=\"viewport\"\u003e\r\n\r\n    \u003ctitle\u003e홈페이지 로그인\u003c/title\u003e\r\n    \r\n    \u003clink href=\"css/basic_root.css\" rel=\"stylesheet\"\u003e\r\n\r\n\r\n    \u003clink href=\"bootstrap-5.0.2-dist/css/bootstrap.css\" rel=\"stylesheet\"\u003e\r\n\r\n    \u003c!-- fontawesome--\u003e\r\n     \u003clink href=\"js/fontawesome-6.3.0/css/all.css\" rel=\"stylesheet\"\u003e\r\n\r\n    \u003cscript src=\"bootstrap-5.0.2-dist/js/bootstrap.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\r\n\r\n    \u003c!-- jquery --\u003e\r\n    \u003cscript src=\"js/jquery-3.6.4.js\"\u003e\u003c/script\u003e\r\n\r\n    \u003c!-- datatables --\u003e\r\n    \u003cscript src=\"js/datatables/datatables.js\"\u003e\u003c/script\u003e\r\n    \u003clink href=\"js/datatables/Select-1.6.1/css/select.dataTables.css\" rel=\"stylesheet\"\u003e\r\n    \u003cscript src=\"js/datatables/DataTables-1.13.3/js/dataTables.bootstrap5.js\"\u003e\u003c/script\u003e\r\n    \u003cscript src=\"js/datatables/Buttons-2.3.5/js/dataTables.buttons.js\"\u003e\u003c/script\u003e\r\n    \u003clink href=\"js/datatables/Buttons-2.3.5/css/buttons.dataTables.css\" rel=\"stylesheet\"\u003e\r\n\r\n    \u003c!-- jqueryui--\u003e\r\n    \u003cscript src=\"js/jquery-ui/jquery-ui.js\"\u003e\u003c/script\u003e\r\n    \u003clink href=\"js/jquery-ui/jquery-ui.css\" rel=\"stylesheet\"\u003e\r\n\r\n\t\u003c!--커스텀 css --\u003e\r\n    \u003c!-- datatable custom css --\u003e\r\n    \u003clink href=\"css/datatable_custom.css\" rel=\"stylesheet\"\u003e\r\n\r\n    \u003c!-- calendar custom css --\u003e\r\n    \u003clink href=\"css/custom.css\" rel=\"stylesheet\"\u003e\r\n\r\n    \u003cscript src=\"js/common.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\r\n\r\n    \u003c!-- 메인 css 파일 --\u003e\r\n\t\u003clink href=\"css/style.css\" rel=\"stylesheet\"\u003e\r\n    \u003clink href=\"css/main.css\" rel=\"stylesheet\"\u003e\r\n\t\u003clink href=\"css/login.css\" rel=\"stylesheet\"\u003e\r\n    \u003clink href=\"css/top_icon.css\" rel=\"stylesheet\"\u003e\r\n\t\r\n\t\u003c!-- 커스텀 js --\u003e\r\n\r\n    \u003cscript src=\"js/order_table.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\r\n\r\n    \u003cscript src=\"js/main.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\r\n\r\n    \u003cscript src=\"js/modal.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\r\n\u003c/head\u003e\r\n\u003cbody id=\"\" class=\"body-login\"\u003e\r\n    \u003cdiv class=\"wrapper\"\u003e\u003c!-- 전체영역(wrapper) --\u003e\r\n        \u003cdiv class=\"container\"\u003e\u003c!-- 전체영역(container) --\u003e\r\n\t\t\t\u003cdiv class=\"content\"\u003e\r\n\t\t\t\t\u003c!-- 로그인 폼 --\u003e\r\n\t\t\t\t\u003cdiv class=\"d-flex flex-column login-area\"\u003e\r\n\t\t\t\t\t\u003c!-- 로그인 타이틀이미지 --\u003e\r\n\t\t\t\t\t\u003c!-- //로그인 타이틀이미지 --\u003e\r\n\t\t\t\t\t\u003c!-- 로그인 내용 --\u003e\r\n\t\t\t\t\t\u003cdiv class=\"login-detail\"\u003e\r\n\t\t\t\t\t\t\u003cdiv class=\"login-degail-group\"\u003e\r\n\t\t\t\t\t\t\t\u003cdiv class=\"login-str\"\u003eLogin\u003c/div\u003e\r\n\t\t\t\t\t\t\t\u003cdiv class=\"login-content\"\u003e\r\n\t\t\t\t\t\t\t\t\u003cul\u003e\r\n\t\t\t\t\t\t\t\t\t\u003cli\u003e\u003cinput type=\"text\" class=\"form-control\" name=\"uid\" placeholder=\"ID\"\u003e\u003c/li\u003e\r\n\t\t\t\t\t\t\t\t\t\u003cli\u003e\u003cinput type=\"password\" class=\"form-control \" name=\"upw\" placeholder=\"Password\"\u003e\u003c/li\u003e\r\n\t\t\t\t\t\t\t\t\t\u003cli\u003e\u003cbutton class=\"login_btn\" type=\"button\" \u003e\u003ci class=\"fa-solid fa-right-to-bracket\"\u003e\u003c/i\u003e 로그인\u003c/button\u003e\u003c/li\u003e\r\n\t\t\t\t\t\t\t\t\t\u003cli\u003e\u003clabel class=\"login-ck mr-10\"\u003e\u003cinput class=\"form-check-input mr-5\" type=\"checkbox\" value=\"\" \u003e\u003cspan\u003eID 저장\u003c/span\u003e\u003c/label\u003e\r\n\t\t\t\t\t\t\t\t\t\u003clabel class=\"login-ck\"\u003e\u003cinput class=\"form-check-input mr-5\" type=\"checkbox\" value=\"\" \u003e\u003cspan\u003e자동로그인\u003c/span\u003e\u003c/label\u003e\u003c/li\u003e\r\n\t\t\t\t\t\t\t\t\u003c/ul\u003e\r\n\t\t\t\t\t\t\t\u003c/div\u003e\r\n\t\t\t\t\t\t\t\u003cdiv class=\"login-etc\"\u003e\r\n\t\t\t\t\t\t\t\t\u003cbutton class=\"login_1_btn mr-10\" type=\"button\" \u003e\u003ci class=\"fa-solid fa-user mr-10\"\u003e\u003c/i\u003e회원가입\u003c/button\u003e\r\n\t\t\t\t\t\t\t\t\u003cbutton class=\"login_1_btn\" type=\"button\" \u003e\u003ci class=\"fa-solid fa-magnifying-glass mr-10\"\u003e\u003c/i\u003e비밀번호찾기\u003c/button\u003e\r\n\t\t\t\t\t\t\t\u003c/div\u003e\r\n\t\t\t\t\t\t\u003c/div\u003e\r\n\t\t\t\t\t\u003c/div\u003e\r\n\t\t\t\t\t\u003c!-- //로그인 내용 --\u003e\r\n\t\t\t\t\u003c/div\u003e\r\n\t\t\t\t\u003c!-- //로그인 폼 --\u003e\r\n\t\t\t\u003c/div\u003e\r\n\t\t\u003c/div\u003e\r\n\t\u003c/div\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e","body_sha256":"5ad54b89e9a7485603e6cba22ae735063fdcfae317d8700956c09f87ecff1dc1","body_length":3519,"content_length":-1,"request":{"url":{"scheme":"http","host":"178.16.143.250","path":"/webpage/index.html"},"method":"GET","headers":{"accept":["*/*"],"referer":["http://178.16.143.250/"],"user_agent":["Mozilla/5.0 zgrab/0.x"]}}},"redirect_response_chain":[{"status_line":"302 Found","status_code":302,"protocol":{"name":"HTTP/1.1","major":1,"minor":1},"headers":{"cache_control":["no-store, no-cache, must-revalidate"],"content_length":["0"],"content_type":["text/html; charset=UTF-8"],"date":["Tue, 11 Mar 2025 18:03:29 GMT"],"expires":["Thu, 19 Nov 1981 08:52:00 GMT"],"location":["./webpage/index.html"],"pragma":["no-cache"],"server":["Apache/2.4.58 (Ubuntu)"],"set_cookie":["PHPSESSID=lsahb3p2j8sa5cbbadkn9avqlb; path=/"]},"headers_raw":"SFRUUC8xLjEgMzAyIEZvdW5kDQpEYXRlOiBUdWUsIDExIE1hciAyMDI1IDE4OjAzOjI5IEdNVA0KU2VydmVyOiBBcGFjaGUvMi40LjU4IChVYnVudHUpDQpTZXQtQ29va2llOiBQSFBTRVNTSUQ9bHNhaGIzcDJqOHNhNWNiYmFka245YXZxbGI7IHBhdGg9Lw0KRXhwaXJlczogVGh1LCAxOSBOb3YgMTk4MSAwODo1MjowMCBHTVQNCkNhY2hlLUNvbnRyb2w6IG5vLXN0b3JlLCBuby1jYWNoZSwgbXVzdC1yZXZhbGlkYXRlDQpQcmFnbWE6IG5vLWNhY2hlDQpMb2NhdGlvbjogLi93ZWJwYWdlL2luZGV4Lmh0bWwNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD1VVEYtOA0KDQo=","request":{"url":{"scheme":"http","host":"178.16.143.250","path":"/"},"method":"GET","headers":{"accept":["*/*"],"user_agent":["Mozilla/5.0 zgrab/0.x"]},"host":"178.16.143.250"}}]},"timestamp":"2025-03-11T11:03:29-07:00"}}}
INFO[0000] finished grab at 2025-03-11T11:03:29-07:00
{"statuses":{"http":{"successes":1,"failures":0}},"start":"2025-03-11T11:03:29-07:00","end":"2025-03-11T11:03:29-07:00","duration":"78.730667ms"}

Notes & Caveats

If necessary, explain the motivation for this PR, and note any caveats that apply to your changes or future work that will be needed.

Issue Tracking

Resolves #463