Possible to send a header with `docker run -i ... typesense/docsearch-scraper` ?

[paulrudy]

I'm running typesense on fly.io, and using a github action to run typesense-docsearch-scraper via docker run -i ... typesense/docsearch-scraper.

My typesense app runs on a couple of VM's, which are not an HA cluster, but operate as single-node clusters. I'd like to be able to target a specific VM when I run the scraper. On fly, the way to do that is to send a header with the request: Fly-Prefer-Region: <region-code>

Is there a method to include such a header when running the scraper?

[jasonbosco]

The scraper doesn't have a way to pass in custom headers. But happy to accept a PR that adds support for this.

[paulrudy]

Thanks. It's far above my ability to code a PR for that functionality, but perhaps headers can be passed in as parameters in a query string added to the start_urls and end_urls in the scraper config?

[comatory]

I attempted to create this feature: #76

I'd welcome a review and some guidance. Thank you.