v1.18: [docs] translation support (backport of #35166) (#35169)

[docs] translation support (#35166)

* chore: update crowdin and add serve command

* feat: upload all desired files to crowdin

* fix: absolute urls for pdfs

* fix: do not import components with relative paths

* feat: updated readme

* fix: whitespace

(cherry picked from commit 8f7fda8)

Co-authored-by: Nick Frostbutter <[email protected]>

Author: mergify[bot]

docs/README.md

@@ -63,21 +63,58 @@ npm run start
 
 ## Translations
 
-Translations are sourced from [Crowdin](https://docusaurus.io/docs/i18n/crowdin) and generated when `master` is built.
-For local development use the following two commands in this `docs` directory.
+Translations are sourced from [Crowdin](https://docusaurus.io/docs/i18n/crowdin)
+and generated when the branch noted as the `STABLE` channel is built via the
+`build.sh` script.
+
+For local development, and with the `CROWDIN_PERSONAL_TOKEN` env variable set,
+use the following two commands in this `docs` directory.
 
 To download the newest documentation translations run:
 
 ```sh
 npm run crowdin:download
 ```
 
-To upload changes from `src` & generate [explicit IDs](https://docusaurus.io/docs/markdown-features/headings#explicit-ids):
+To upload changes from `src` & generate
+[explicit IDs](https://docusaurus.io/docs/markdown-features/headings#explicit-ids):
 
 ```shell
 npm run crowdin:upload
 ```
 
+> Translations are only included when deploying the `STABLE` channel of the docs
+> (via `build.sh`). Resulting in only the `docs.solanalabs.com` documentation
+> site to include translated content. Therefore, the `edge` and `beta` docs
+> sites are not expected to include translated content, even though the language
+> selector will still be present.
+
+### Common issues
+
+#### `CROWDIN_PERSONAL_TOKEN` env variable
+
+The `crowdin.yml` file requires a `CROWDIN_PERSONAL_TOKEN` env variable to be
+set with a valid Crowdin access token.
+
+For local development, you can store this in a `.env` file that the Crowdin CLI
+will auto detect.
+
+For building and publishing via the GitHub actions, the `CROWDIN_PERSONAL_TOKEN`
+secret must be set.
+
+#### Translation locale fails to build with `SyntaxError`
+
+Some translation locales may fail to build with a `SyntaxError` thrown by
+Docusaurus due to how certain language symbols get parsed by Docusaurus while
+generating the static version of the docs.
+
+> Note: When any locale fails to build, the entire docs build will fail
+> resulting in the docs not being able to be deployed at all.
+
+There are several known locales that fail to build the current documentation.
+They are listed in the commented out `localesNotBuilding` attribute in the
+[`docusaurus.config.js`](https://github.com/solana-labs/solana/blob/master/docs/docusaurus.config.js)
+
 ## CI Build Flow
 
 The docs are built and published in Travis CI with the `./build.sh` script. On each PR, the docs are built, but not published.

docs/crowdin.yml

@@ -4,13 +4,22 @@ base_url: 'https://solana.crowdin.com'
 preserve_hierarchy: true
 files: [
     # JSON translation files
-    # {
-    # source: '/i18n/en/**/*',
-    # translation: '/i18n/%two_letters_code%/**/%original_file_name%',
-    #},
+    {
+    source: '/i18n/en/**/*',
+    translation: '/i18n/%two_letters_code%/**/%original_file_name%',
+    },
     # Docs Markdown files
     {
       source: '/src/**/*.md',
       translation: '/i18n/%two_letters_code%/docusaurus-plugin-content-docs/current/**/%original_file_name%',
     },
+    {
+      source: '/src/**/*.mdx',
+      translation: '/i18n/%two_letters_code%/docusaurus-plugin-content-docs/current/**/%original_file_name%',
+    },
+    # Custom sidebar category files
+    {
+      source: '/src/**/*.json',
+      translation: '/i18n/%two_letters_code%/docusaurus-plugin-content-docs/current/**/%original_file_name%',
+    },
   ]

docs/package-lock.json

@@ -5,6 +5,7 @@
   "scripts": {
     "start": "docusaurus start",
     "build": "docusaurus build",
+    "serve": "docusaurus serve",
     "clear": "docusaurus clear",
     "help": "docusaurus --help",
     "swizzle": "docusaurus swizzle",
@@ -20,7 +21,7 @@
     "crowdin:upload": "npm run write-i18n && crowdin upload"
   },
   "dependencies": {
-    "@crowdin/cli": "^3.6.1",
+    "@crowdin/cli": "^3.17.0",
     "@docusaurus/core": "^2.2.0",
     "@docusaurus/plugin-google-gtag": "^2.4.0",
     "@docusaurus/preset-classic": "^2.2.0",

docs/package.json

@@ -55,6 +55,6 @@ Explore what it takes to operate a Solana validator and help secure the network.
 
 ## Learn more
 
-import HomeCtaLinks from "../components/HomeCtaLinks";
+import HomeCtaLinks from "@site/components/HomeCtaLinks";
 
 <HomeCtaLinks />

docs/src/index.mdx

@@ -41,7 +41,8 @@ cannot change the original value that is contained in a commitment.
 Interested readers can refer to the following resources for a more in-depth
 treatment of Pedersen commitment and the (twisted) ElGamal encryption schemes.
 
-- [Notes](./zk-docs/twisted_elgamal.pdf) on the twisted ElGamal encryption
+- [Notes](https://github.com/solana-labs/solana/blob/master/docs/src/runtime/zk-docs/twisted_elgamal.pdf)
+  on the twisted ElGamal encryption
 - A technical
   [overview](https://github.com/solana-labs/solana-program-library/blob/master/token/zk-token-protocol-paper/part1.pdf)
   of the SPL Token 2022 confidential extension
@@ -98,14 +99,14 @@ The ZK Token proof program supports the following list of zero-knowledge proofs.
   - The ElGamal public-key validity proof instruction certifies that an ElGamal
     public-key is a properly formed public key.
   - Mathematical description and proof of security:
-    [[Notes]](./zk-docs/pubkey_proof.pdf)
+    [[Notes]](https://github.com/solana-labs/solana/blob/master/docs/src/runtime/zk-docs/pubkey_proof.pdf)
 
 - `VerifyZeroBalance`:
 
   - The zero-balance proof certifies that an ElGamal ciphertext encrypts the
     number zero.
   - Mathematical description and proof of security:
-    [[Notes]](./zk-docs/zero_proof.pdf)
+    [[Notes]](https://github.com/solana-labs/solana/blob/master/docs/src/runtime/zk-docs/zero_proof.pdf)
 
 #### Equality proofs
 
@@ -114,11 +115,11 @@ The ZK Token proof program supports the following list of zero-knowledge proofs.
   - The ciphertext-commitment equality proof certifies that an ElGamal
     ciphertext and a Pedersen commitment encode the same message.
   - Mathematical description and proof of security:
-    [[Notes]](./zk-docs/ciphertext_commitment_equality.pdf)
+    [[Notes]](https://github.com/solana-labs/solana/blob/master/docs/src/runtime/zk-docs/ciphertext_commitment_equality.pdf)
 
 - `VerifyCiphertextCiphertextEquality`:
 
   - The ciphertext-ciphertext equality proof certifies that two ElGamal
     ciphertexts encrypt the same message.
   - Mathematical description and proof of security:
-    [[Notes]](./zk-docs/ciphertext_ciphertext_equality.pdf)
+    [[Notes]](https://github.com/solana-labs/solana/blob/master/docs/src/runtime/zk-docs/ciphertext_ciphertext_equality.pdf)