Skip to content

Commit 14474b8

Browse files
sirutBuasaisirutBuasai
authored
Gen release (#5596)
* add reference index.md Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update steering Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update available_images.md generation config files Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update available_images.md generation config files Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update available_images.md generation config files Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * excempt templates Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add legacy data Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add legacy data Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add doc annotations Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update documentation Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix legacy support comment Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
1 parent 127cb41 commit 14474b8

File tree

210 files changed

+3476
-1978
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

210 files changed

+3476
-1978
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ __pycache__
55
.venv
66
.ruff_cache
77
.pytest_cache
8+
9+
# Generated docs
810
docs/tutorials
911
docs/reference/available_images.md
1012
docs/reference/support_policy.md

‎.kiro/steering/docs.md‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,251 @@
22

33
When working in `deep-learning-containers` repository under the directory `docs`, you are actively working on writing documentations for AWS Deep Learning Containers.
44

5+
## Project guideline
6+
57
Our documentations uses Material for MkDocs lightweight framework for all our static documentations. When working, ensure that we are using industry best practices when organizing the codebase unless there are any specific instructions provided.
68

79
Within this documentations page, website navigation through `.nav.yml` file should be in one file at `docs/.nav.yml` unless a developer specify otherwise. This is done so that we have a central place to organize our pages.
10+
11+
## Documentation Generation System
12+
13+
The documentation uses an automatic generation system for `available_images.md` and `support_policy.md`.
14+
15+
### Directory Structure
16+
17+
```
18+
docs/src/
19+
├── templates/reference/ # Jinja2 templates
20+
│ ├── available_images.template.md
21+
│ └── support_policy.template.md
22+
├── tables/ # Table column configs (one per repository)
23+
│ ├── pytorch-training.yml
24+
│ ├── pytorch-inference.yml
25+
│ └── ...
26+
├── data/ # Image configs (one file per image)
27+
│ ├── pytorch-training/
28+
│ │ ├── 2.9-gpu-ec2.yml
29+
│ │ ├── 2.9-cpu-sagemaker.yml
30+
│ │ └── ...
31+
│ ├── pytorch-training-neuronx/
32+
│ │ └── 2.9-neuronx-sdk2.27.1.yml
33+
│ └── ...
34+
├── legacy/ # Historical support data
35+
│ └── legacy_support.yml
36+
├── constants.py # Path constants and global variables
37+
├── generate.py # Generation logic
38+
├── global.yml # Shared terminology and configuration
39+
├── hooks.py # MkDocs hooks
40+
├── logger.py # Logging configuration
41+
├── macros.py # MkDocs macros
42+
├── main.py # CLI entry point
43+
└── utils.py # Reusable helper functions
44+
```
45+
46+
### Adding a New Image
47+
48+
1. Create a new YAML file in `docs/src/data/<repository>/`:
49+
50+
- Naming: `<version>-<accelerator>-<platform>.yml`
51+
- For Neuron: `<version>-neuronx-sdk<sdk_version>.yml`
52+
53+
1. Required fields:
54+
55+
```yaml
56+
framework: PyTorch # Display name
57+
version: "2.9" # Framework version
58+
accelerator: gpu # gpu, cpu, or neuronx
59+
python: py312 # Python version
60+
platform: ec2 # ec2 or sagemaker
61+
tag: "2.9.0-gpu-py312-cu130-ubuntu22.04-ec2" # Full image tag
62+
```
63+
64+
1. Optional fields:
65+
66+
```yaml
67+
cuda: cu130 # For GPU images
68+
sdk: "2.27.1" # For Neuron images
69+
os: ubuntu22.04 # Operating system
70+
public_registry: true # If in ECR Public Gallery
71+
ga: "2025-10-15" # GA date (for support policy)
72+
eop: "2026-10-15" # EOP date (for support policy)
73+
ecr_account: "007439368137" # Override default ECR account
74+
```
75+
76+
1. GA/EOP dates are only needed for repositories that appear in support policy (base, pytorch-*, tensorflow-* excluding neuronx).
77+
78+
### Adding a New Repository
79+
80+
1. Create directory: `docs/src/data/<repository-name>/`
81+
82+
1. Create table config: `docs/src/tables/<repository-name>.yml`
83+
84+
```yaml
85+
columns:
86+
- field: framework_version
87+
header: "Framework"
88+
- field: python
89+
header: "Python"
90+
# ... add columns in desired order
91+
```
92+
93+
1. Add to `docs/src/global.yml`:
94+
95+
- Add display name to `display_names` (required, error if missing)
96+
- Add to `table_order` list
97+
98+
### Global Configuration (global.yml)
99+
100+
Uses OmegaConf for variable interpolation with `${var}` syntax:
101+
102+
```yaml
103+
# Base terminology
104+
aws: "AWS"
105+
amazon: "Amazon"
106+
dlc: "Deep Learning Containers"
107+
108+
# Composed terminology (resolved at load time)
109+
dlc_long: "${aws} ${dlc}" # -> "AWS Deep Learning Containers"
110+
sagemaker: "${amazon} SageMaker" # -> "Amazon SageMaker"
111+
112+
# Platform display mappings
113+
platforms:
114+
ec2: "EC2, ECS, EKS"
115+
sagemaker: "SageMaker"
116+
117+
# Repository display names (required for all repositories)
118+
display_names:
119+
pytorch-training: "PyTorch Training"
120+
# ...
121+
122+
# Framework groups for support policy consolidation (lowercase keys)
123+
framework_groups:
124+
pytorch:
125+
- pytorch-training
126+
- pytorch-inference
127+
- pytorch-training-arm64
128+
- pytorch-inference-arm64
129+
tensorflow:
130+
- tensorflow-training
131+
- tensorflow-inference
132+
- tensorflow-inference-arm64
133+
134+
# Table order (controls order in available_images.md and support_policy.md)
135+
table_order:
136+
- base
137+
- pytorch-training
138+
# ...
139+
```
140+
141+
### Support Policy Consolidation
142+
143+
The `framework_groups` configuration consolidates support policy rows by framework. Repositories in the same group are combined into a single row using the framework name (e.g., "PyTorch").
144+
145+
**Requirements:**
146+
147+
- All repositories in a group that have a given version must have identical GA/EOP dates
148+
- Missing versions in some repositories are allowed (only present repos are consolidated)
149+
- A `ValueError` is raised if dates differ within a group for the same version
150+
151+
To add a new framework group, add an entry to `framework_groups` with the framework name as key and list of repositories as value.
152+
153+
### Reordering Tables and Columns
154+
155+
- **Table order**: Modify `table_order` list in `global.yml`
156+
- **Column order**: Modify `columns` list order in `tables/<repository>.yml`
157+
158+
### Image Sorting in Tables
159+
160+
Images in `available_images.md` are automatically sorted by:
161+
162+
1. **Version** (descending) - newest versions first
163+
1. **Platform** - SageMaker before EC2
164+
1. **Accelerator** - GPU before NeuronX before CPU
165+
166+
### Running Generation
167+
168+
```bash
169+
# Development
170+
cd docs && source .venv/bin/activate
171+
cd src && python main.py --verbose
172+
173+
# With MkDocs (automatic via hooks)
174+
mkdocs serve
175+
mkdocs build
176+
```
177+
178+
### Support Policy Logic
179+
180+
- Scans all image configs for `ga` and `eop` fields
181+
- Groups by (repository, version)
182+
- Validates consistency within each group
183+
- Auto-determines supported/unsupported by comparing `eop` to current date
184+
- Uses `display_names` for Framework column
185+
186+
### Legacy Support Data
187+
188+
Historical support policy data for older, unsupported images is stored in `docs/src/legacy/legacy_support.yml`.
189+
190+
#### Directory Structure
191+
192+
```
193+
docs/src/legacy/
194+
└── legacy_support.yml # Historical support data
195+
```
196+
197+
#### File Format
198+
199+
```yaml
200+
PyTorch:
201+
- version: "2.5"
202+
ga: "2024-10-29"
203+
eop: "2025-10-29"
204+
- version: "2.4"
205+
ga: "2024-07-24"
206+
eop: "2025-07-24"
207+
```
208+
209+
#### Adding Legacy Entries
210+
211+
1. Open `docs/src/legacy/legacy_support.yml`
212+
1. Add entries under the framework name key (must match `display_names` keys for framework groups, e.g., `pytorch`, `tensorflow`)
213+
1. Each entry needs: `version`, `ga`, `eop`
214+
215+
#### Behavior
216+
217+
- Legacy entries appear only in `support_policy.md` (unsupported section)
218+
- Images past their EOP date are automatically filtered from `available_images.md`
219+
- The `is_image_supported()` function checks if `eop >= today`
220+
221+
## Code Organization
222+
223+
### File Responsibilities
224+
225+
- `generate.py` - Contains only `generate_*` functions for documentation generation
226+
- `utils.py` - Contains all reusable helper functions
227+
- `macros.py` - MkDocs macros plugin integration
228+
- `hooks.py` - MkDocs hooks entry point
229+
- `constants.py` - All path constants and global variables
230+
231+
### utils.py Conventions
232+
233+
All functions in `utils.py` must:
234+
235+
1. Have a docstring explaining what the function does
236+
1. Be fully typed with type hints
237+
1. Document exceptions raised (if any)
238+
239+
Example:
240+
241+
```python
242+
def get_display_name(global_config: dict, repository: str) -> str:
243+
"""Get human-readable display name for a repository.
244+
245+
Raises:
246+
KeyError: If repository not found in global config display_names.
247+
"""
248+
```
249+
250+
## Update knowledge base
251+
252+
If there are any new changes to the documentations generation and organization, make sure to update you knowledge base in the steering/docs.md file.

‎.pre-commit-config.yaml‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,7 @@ repos:
4343
name: "📁 filesystem/🔧 config · Validate YAML file"
4444
- id: detect-aws-credentials
4545
name: "🔒 security · Detect aws credentials"
46-
args:
47-
- --allow-missing-credentials
46+
args: ["--allow-missing-credentials"]
4847
- id: detect-private-key
4948
name: "🔒 security · Detect private keys"
5049

@@ -63,7 +62,7 @@ repos:
6362
hooks:
6463
- id: typos
6564
name: "✏️ typos · Checking typos"
66-
args: [--force-exclude]
65+
args: ["--force-exclude"]
6766

6867

6968
### Python Tools ###
@@ -97,17 +96,20 @@ repos:
9796

9897

9998
### Markdown ###
100-
- repo: https://github.com/hukkin/mdformat
101-
rev: 1.0.0
99+
- repo: local
102100
hooks:
103-
- id: mdformat
101+
- id: flowmark
104102
name: "📝 markdown · Format markdown (docs)"
103+
entry: flowmark --inplace --nobackup -w 150
104+
language: python
105+
additional_dependencies: [flowmark]
105106
files: ^docs/
106-
additional_dependencies:
107-
- mdformat-gfm
108-
- mdformat-mkdocs[recommended]
109-
- mdformat-ruff
110-
- ruff
107+
exclude: (\.template\.md$|/\.venv/)
108+
types: [markdown]
109+
110+
- repo: https://github.com/hukkin/mdformat
111+
rev: 1.0.0
112+
hooks:
111113
- id: mdformat
112114
name: "📝 markdown · Format markdown"
113115
exclude: ^docs/
@@ -122,9 +124,7 @@ repos:
122124
hooks:
123125
- id: dockerfmt
124126
name: "🐳 docker · Format docker"
125-
args:
126-
- --indent=2
127-
- --write
127+
args: ["--indent", "2", "--write"]
128128

129129

130130
### Shell ###

0 commit comments

Comments
 (0)