Modules that make auxiliary requests should emit HTTP_RESPONSE

[Sh4d0wHunt3rX]

I'm testing bypass403 module on my own site. I have a question.

For example from cloudflare I added this header rule, so it will work like this:

curl -ks -H 'X-Forwarded-For: 127.0.0.1' -X GET 'https://www.3r.wtf/' -H 'User-Agent: Mozilla/5.0'

Now, I use this:

bbot -t 3r.wtf -m httpx bypass403 -om asset_inventory -c web_spider_distance=4 web_spider_depth=4

Now, I expect bbot to actually bypass it and crawl the links, but it's not crawling.

Can't it automatically add the header to requests to bypass and crawl?

[liquidsec]

I think we could probably solve this by manually emitting the contents of the successful bypass as an http_response, @TheTechromancer do you see any issues with doing that?

@Amiremami would you be ok with me testing using your site you posted if we do this?

[Sh4d0wHunt3rX]

Yes of course, it's ok by me 🙏

[TheTechromancer]

It would make sense to add this feature after we've implemented the Web Engine. That will include a helper that returns the response in a JSON format similar to HTTP_RESPONSE.

[liquidsec]

adding on-hold tag

[liquidsec]

@TheTechromancer does it make sense to make that helper yet?

[TheTechromancer]

Yeah I think now is a good time

[liquidsec]

bump

[liquidsec]

This is possible now.

The first implementation of this is present in #2967

There needs to be a review of what other modules could benefit