|
1 | 1 | # Site search on ethereum.org |
2 | 2 |
|
3 | | -TL;DR: we use Algolia to implement a site search feature on ethereum.org. |
| 3 | +TL;DR: we use Algolia to implement a site search feature on ethereum.org. As an open source project, Algolia has sponsored the crawling and indexing of the entire site. |
4 | 4 |
|
5 | | -## What do we use Algolia and Docsearch for? |
| 5 | +## What do we use Algolia and DocSearch for? |
6 | 6 |
|
7 | | -Algolia allows us to index the content on ethereum.org and implement a powerful site search tool on ethereum.org. In order to create the index of our content, we use a web crawling tool called Docsearch. Docsearch takes a start_urls of ethereum.org and crawls the site to index the content based on a [docsearchConfig file](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearchConfig.json). |
| 7 | +Algolia allows us to index the content on ethereum.org and implement a powerful site search tool on ethereum.org. In order to create the index of our content, we use a web crawling tool called DocSearch. DocSearch takes a starting URL of ethereum.org and crawls the site to index the content, based on a custom configuration setup held with the service. |
8 | 8 |
|
9 | | -We kick off the crawling and indexing of ethereum.org through a GitHub Action that triggers on the merge to `master` branch. [View the GitHub Action](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearch-crawl.yml). |
| 9 | +Site crawling and indexing is performed by default on a weekly basis on Friday afternoons. This is performed automatically by Algolia servers, which scrape the entire production site of ethereum.org to build an index. This index is hosted by Algolia for use on the site. |
10 | 10 |
|
11 | | -## Docsearch Config |
| 11 | +## DocSearch Config |
12 | 12 |
|
13 | | -Some important notes about the docsearch config file: |
| 13 | +Some important notes about the DocSearch config: |
14 | 14 |
|
15 | 15 | ### Configuration |
16 | 16 |
|
17 | | -- `index_name` is the name of the algolia index where the generated index will be uploaded to. |
18 | | -- `start_urls` are the urls that the crawler will start from. Some important attributes in the `start_urls` that we use are: |
19 | | - - `lang`: regex path to different languages that the site is translated to that need crawling. Since ethereum.org is translated to 37+ languages, we need to be able to crawl the website in each language for indexing. |
20 | | - - `page_rank`: the rank of pages that breaks ties when multiple query results have the same weight. This weight is derived from the selectors. |
21 | | -- `stop_urls` is used to strip out query parameters in the websites urls. We were running into issues where we were getting duplicate query results due to query parameters making urls unique. Stripping these out solved our deduplication problem. |
22 | | -- selectors are used to specify what the crawler should look for when weighting content for the index. |
23 | | - |
24 | | -### Generation |
25 | | - |
26 | | -We generate the docsearchConfig.json file using a [script](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearchConfigScript.js). This allows us to dynamically pull in the languages the websites support from the [translations.json data file](https://github.com/ethereum/ethereum-org-website/blob/dev/src/data/translations.json). Our GitHub action executes this script. |
| 17 | +- `indexName` is the name of the Algolia index where the generated index will be uploaded to |
| 18 | +- `startUrls` are the urls that the crawler will start from |
| 19 | +- Translated pages are automatically faceted for search results based on the `<html lang="">` attribute of each page |
| 20 | +- Selectors are used to specify what the crawler should look for when weighting content for the index. |
| 21 | +- CheerioAPI can be utilized within the crawler using the `$` selector to manipulate the DOM before indexing each page |
| 22 | +- Elements to be ignored are removed before indexing using the CheerioAPI library: `$('selector').remove()`. This includes `aside`, `nav`, `footer` and `style` elements. |
| 23 | +- While building pages, semantic naming with the aforementioned elements, i.e. `aside`, will ignore any content contained within. This is beneficial for content that is not directly related to the page content, such as callouts, banners, quiz content, or navigation elements. |
27 | 24 |
|
28 | 25 | ## Resources |
29 | 26 |
|
30 | 27 | - [Algolia documentation](https://www.algolia.com/doc/) |
31 | | -- [Docsearch documentation](https://docsearch.algolia.com/docs/what-is-docsearch) |
32 | | -- [Docsearch scraper Docker image](https://hub.docker.com/r/algolia/docsearch-scraper) |
| 28 | +- [DocSearch documentation](https://docsearch.algolia.com/docs/what-is-docsearch) |
0 commit comments