docs: instruction for scraping without running on port 80 #372
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@jasonbosco hi!! Is the PR needed or not really? |
jasonbosco
requested changes
Sep 9, 2025
| Ideally, you will need to run your site at port `:80`, because the scraper can present a not expected behavior if has hosted in another port. | ||
| However, there is an option to scrape your site at port other than `:80` by specifying `"allowed_domains": ["localhost"]` in yours scraper configuration. | ||
| Then it is possible to write `"start_urls": ["http://localhost:<your-port>"]`. | ||
| More detailed example of the configuration can be found [here](https://github.com/meilisearch/docs-scraper/issues/103#issuecomment-810736674) |
There was a problem hiding this comment.
The instructions in this guide assume that the scraper is being run from within a Docker container, and that the website is running somewhere outside the docker container. Which is why we have a mention of the URL to use in the paragraph above this.
Using localhost in the scraper configuration when it is running inside a Docker container will make the scraper look for the docs site to also be running inside the same Docker container and won't work.
So the instructions here, won't work with the way the rest of the guide is structured around running the scraper as a Docker container.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Summary
Added a short instruction on how to enable typesense-docs-search scrapper to scrape website launched on port different from
:80.I have chosen to update here since it is a starting point for any docs search setup.
PR Checklist