Merge pull request #2272 from melissawm:contributing-docs

PiperOrigin-RevId: 807369031

Author: Google-ML-Automation

docs/development.md

@@ -0,0 +1,38 @@
+```{include} ../CONTRIBUTING.md
+```
+
+## Contributing documentation
+
+The MaxText documentation website is built using [Sphinx](https://www.sphinx-doc.org) and [MyST](https://myst-parser.readthedocs.io/en/latest/). Documents are written in [MyST Markdown syntax](https://myst-parser.readthedocs.io/en/latest/syntax/typography.html#syntax-core).
+
+### Building the documentation locally (optional)
+
+If you are writing documentation for MaxText, you may want to preview the documentation site locally to ensure things work as expected before a deployment to Read The Docs.
+
+First, make sure you install the necessary dependencies. You can do this by navigating to your local clone of the MaxText repo and running:
+
+```bash
+pip install -r requirements_docs.txt
+```
+
+Once the dependencies are installed, you can navigate to the `docs/` folder and run:
+
+```bash
+sphinx-build -b html . _build/html
+```
+
+This will generate the documentation in the `docs/_build/html` directory. These files can be opened in a web browser directly, or you can use a simple HTTP server to serve the files. For example, you can run:
+
+```bash
+python -m http.server -d docs/_build/html
+```
+
+Then, open your web browser and navigate to `http://localhost:8000` to view the documentation.
+
+### Adding new documentation files
+
+If you are adding a new document, make sure it is included in the `toctree` directive corresponding to the section where the new document should live. For example, if adding a new tutorial, make sure it is listed in [the `docs/tutorials.md`](https://github.com/AI-Hypercomputer/maxtext/blob/7070e8eecbea8951c8e5281219ce797c8df1441f/docs/tutorials.md?plain=1#L38).
+
+### Documentation deployment
+
+The MaxText documentation is deployed to [https://maxtext.readthedocs.io](https://maxtext.readthedocs.io) on any successful merge to the main branch.

docs/guides/data_input_hf.md

@@ -1,6 +1,7 @@
 # Hugging Face pipeline
 The Hugging Face pipeline supports streaming directly from the Hugging Face Hub, or from a Cloud Storage bucket in Hugging Face supported formats (parquet, json, etc.). This is through the Hugging Face [`datasets.load_dataset` API](https://huggingface.co/docs/datasets/en/loading) with `streaming=True`, which takes in `hf_*` parameters.
 ## Example config for streaming from Hugging Face Hub (no download needed):
+
 In `src/MaxText/configs/base.yml` or through command line, set the following parameters:
 ```
 dataset_type: hf
@@ -18,6 +19,7 @@ hf_access_token: ''  # provide token if using gated dataset or tokenizer
 ```
 
 ## Example config for streaming from downloaded data in a Cloud Storage bucket:
+
 In `src/MaxText/configs/base.yml` or through command line, set the following parameters:
 ```
 dataset_type: hf

docs/index.md

@@ -100,4 +100,5 @@ tutorials.md
 guides.md
 explanations.md
 reference.md
+development.md
 ```