Skip to content

Commit 2a69d60

Browse files
JosverlJosverl
committed
docs: Add agent instructions
Signed-off-by: Jos Verlinde <[email protected]>
1 parent 42df90c commit 2a69d60

File tree

2 files changed

+273
-1
lines changed

2 files changed

+273
-1
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
# MicroPython-Stubs Repository Guide
2+
3+
## Project Overview
4+
5+
This repository mostly contains **type stubs**, not executable code.
6+
It hosts 3,000+ stub files (`.pyi`) for MicroPython across multiple versions (v1.17-v1.27), ports (esp32, rp2, stm32, etc.), and boards. These stubs enable IDE autocomplete, type checking, and documentation for MicroPython development.
7+
8+
**Sister Repository**: [micropython-stubber](https://github.com/Josverl/micropython-stubber) - CLI tool (`stubber`) that generates these stubs.
9+
10+
## Repository Structure
11+
12+
```
13+
stubs/ # Generated stub files (MCU, frozen, doc, merged)
14+
publish/ # Ready-to-install PyPI packages
15+
micropython-v1_XX_X-{port}-{board}-stubs/
16+
micropython-stdlib-stubs/
17+
tools/ # Tooling (board_compare)
18+
scripts/ # Analysis scripts and notebooks
19+
docs/ # Sphinx documentation source
20+
data/ # JSON database files (stub-packages.json)
21+
tests/quality_tests/ # Pytest-based stub quality tests
22+
```
23+
24+
## Key Concepts
25+
26+
### Stub Types (Half-Products)
27+
1. **MCU stubs**: Generated on-device, precise but minimal type info (`stubs/micropython-{version}-{port}[-{board}]`)
28+
2. **Frozen stubs**: From manifest files, includes frozen modules (`stubs/micropython-{version}-Frozen/{port}/{board}`)
29+
3. **Doc stubs**: Parsed from MicroPython `.rst` docs, rich typing (`stubs/micropython-{version}-docstubs`)
30+
4. **Merged stubs**: MCU + Doc, precise + rich (`stubs/micropython-{version}-{port}-merged`)
31+
5. **Core stubs**: Manual overrides for problematic modules (`stubs/micropython-core`)
32+
33+
### Naming Convention
34+
`micropython-{flat_version}-{port}-{board}-stubs` where `flat_version` uses underscores (e.g., `v1_24_1`).
35+
36+
## Development Workflows
37+
38+
### Platform Notes
39+
- **Primary**: Windows
40+
- **Secondary**: Linux
41+
- **CI/CD** :GitHub Actions workflows use Ubuntu
42+
43+
### Database
44+
45+
#### `data/all_packages.db`
46+
The project uses a SQLite database (`package_data.db`) to store metadata about packages
47+
and project hashes to detect if a package has changed since it was last published.
48+
49+
This database is generated and maintained by the `micropython-stubber` tool during the publishing process.
50+
It is checked in to the repository to allow scripts and tools to query package information as it cannot be
51+
regenerated easily.
52+
53+
The database can be queried, but should not be modified directly.
54+
There is also a JSON export of the database at `data/all_modules.json` which can be used for read-only queries without needing SQLite.
55+
56+
57+
### Database Access - IMPORTANT
58+
**SQLite3 CLI tools are NOT installed.** When working with `package_data.db` or database queries:
59+
- ✅ **USE**: Data Store MCP servers (available in this environment)
60+
- ❌ **AVOID**: Writing one-off SQLite scripts/commands that require debugging
61+
- Alternative: Query `all_modules.json` (JSON export of package data) or `data/stub-packages.json`
62+
63+
### Core Tools
64+
65+
#### The `stubber` CLI
66+
From [micropython-stubber](https://github.com/Josverl/micropython-stubber), installed via pip:
67+
```powershell
68+
pip install micropython-stubber # or --pre for preview versions
69+
```
70+
71+
**Common Commands**:
72+
```powershell
73+
stubber clone --add-stubs # Clone micropython + micropython-lib repos
74+
stubber get-docstubs --version v1.24.1 # Generate doc stubs
75+
stubber get-frozen --version v1.24.1 # Generate frozen stubs
76+
stubber merge --version v1.24.1 # Merge MCU + doc stubs
77+
stubber build --version v1.24.1 # Build publishable package
78+
```
79+
80+
#### Update Scripts
81+
- **`update_all_modules.py`**: Regenerates `all_modules.json` from `publish/` packages
82+
- **`docs/update_docs.py`**: Updates documentation files
83+
84+
#### Build/Test Tasks
85+
VSCode tasks (`.vscode/tasks.json`):
86+
- `Sphinx: build documentation` - Default build (Ctrl+Shift+B)
87+
- `Sphinx: clean build documentation`
88+
89+
### Typical Development Flow
90+
91+
1. **Update stubs for a version** (using stubber CLI):
92+
```bash
93+
# example: version = v1.24.1
94+
stubber get-docstubs --version v1.24.1
95+
stubber get-frozen --version v1.24.1
96+
stubber merge --version v1.24.1 --port rp2 --board rpi_pico
97+
stubber build --version v1.24.1 --port rp2 --board rpi_pico
98+
```
99+
100+
2. **Interactive development** (Jupyter notebooks):
101+
- `Manual stub build chain.ipynb` - Interactive stub generation workflow
102+
- `scripts/package_db_to_json.ipynb` - Database export operations
103+
- `scripts/find_undoc_funcs.ipynb` - Documentation analysis
104+
105+
3. **Test stub quality**:
106+
```bash
107+
pytest tests/quality_tests/ -v
108+
# Clear cache and retest
109+
pytest tests/quality_tests/ --cache-clear
110+
# Test specific version/port
111+
pytest tests/quality_tests/ -k "v1_24_1 and rp2"
112+
```
113+
114+
4. **Update documentation**:
115+
```bash
116+
python docs/update_docs.py
117+
python update_all_modules.py
118+
```
119+
120+
5. **Build docs**:
121+
Run task: `Sphinx: build documentation`
122+
123+
### GitHub Actions (`.github/workflows/`)
124+
- **`update_stubs.yml`**: Daily stub updates using stubber CLI
125+
- **`test_stub_quality.yml`**: Pyright/mypy validation on snippets (15min runtime)
126+
- **`test_runtime_typing.yml`**: Docker-based runtime tests for typing module
127+
- **`publish_explorer.yml`**: Deploys board explorer app
128+
- **`weekly_automation.yml`**: Maintenance tasks
129+
- **`get-doc-stubs.yml`**: Documentation stub generation
130+
- **`update_docs.yml`**: Sphinx documentation build
131+
132+
## Code Conventions
133+
134+
### Python Style
135+
- **Formatter**: Ruff (line-length: 140) ( black was use - is phased out)
136+
- **Linter**: Ruff (extends Black, adds `PYI` for stubs)
137+
- **Type Checker**: Pyright (primary), mypy (secondary)
138+
- **Target**: Python 3.9+ (stubs must be usable from 3.9)
139+
140+
### Stub-Specific Rules (Ruff)
141+
```toml
142+
# From pyproject.toml - these are EXCEPTIONS for MicroPython stubs:
143+
PYI021 = ignore # Docstrings ARE included (documentation purpose)
144+
PYI011 = ignore # Allow complex defaults (MicroPython-specific)
145+
PYI014 = ignore # Allow actual default values (not just ...)
146+
PYI029 = ignore # __str__/__repr__ useful for MicroPython
147+
```
148+
149+
### File Organization
150+
- Stubs use `.pyi` extension (type stub files)
151+
- Frozen modules may have `.py` source alongside `.pyi`
152+
- Package structure mirrors MicroPython module hierarchy
153+
154+
## Testing Strategy
155+
156+
All tests must use the pytest framework only.
157+
158+
159+
### Quality Tests (`tests/quality_tests/`)
160+
Integration tests validate stub quality using code snippets and type checkers.
161+
162+
**Test Structure**:
163+
- **Parametrized**: Test across versions/ports/boards
164+
- **Stub sources**: `pypi`, `pypi-pre`, `local` (from `publish/`)
165+
- **Snippet folders**: `feat_*/` and `check_*/` directories with test code
166+
- **Caching**: 24-hour cache for stub packages (speed up test runs)
167+
- **Tools**: Uses `uv pip` for fast isolated installs to `typings/` folders
168+
169+
**What tests validate**:
170+
- Type checker output (pyright, mypy)
171+
- `assert_type` verifies inferred types match expectations
172+
- `# type: ignore` comments with `--warn-unused-ignores` ensure errors are caught
173+
- Idiomatic MicroPython code should pass without type errors
174+
175+
### Running Tests
176+
```bash
177+
# All snippet tests
178+
pytest tests/quality_tests/ -m snippets -v
179+
180+
# Clear cache and retest
181+
pytest tests/quality_tests/ --cache-clear
182+
183+
# Specific version/port combination
184+
pytest tests/quality_tests/ -k "v1_24_1 and rp2"
185+
186+
# Single test file
187+
pytest tests/quality_tests/test_snippets.py::test_pyright[local-v1.20.0-stm32-stdlib]
188+
```
189+
190+
### Docker-Based Testing
191+
Runtime validation uses official MicroPython Docker images:
192+
193+
**Example** (`tests/quality_tests/feat_typing/manual_run.ipynb`):
194+
```python
195+
# Test typing module functionality at runtime
196+
image = "micropython/unix:v1.23.0"
197+
!docker run -u 1000 -v {cwd}:/code --rm {image} micropython list_libs.py
198+
```
199+
200+
**Use cases**:
201+
- Validate typing module behavior
202+
- Test MicroPython-specific functionality
203+
- Cross-platform compatibility checks
204+
205+
### Jupyter Notebooks for Testing
206+
Notebooks provide interactive development and testing workflows:
207+
208+
**Key notebooks**:
209+
- **`Manual stub build chain.ipynb`**: Complete stub generation pipeline
210+
- Run `stubber docstubs`, `frozen`, `merge`, `build` commands
211+
- Invalidate pytest cache for testing
212+
- Publish to PyPI (test/production)
213+
214+
- **`tests/quality_tests/feat_typing/manual_run.ipynb`**: Docker test runner
215+
- Execute MicroPython scripts in containers
216+
- Validate runtime typing behavior
217+
218+
- **`scripts/package_db_to_json.ipynb`**: Database operations
219+
- Export package data to JSON
220+
- Query module information
221+
222+
**Benefits of notebooks**:
223+
- Interactive exploration of stub generation
224+
- Quick iteration on test scenarios
225+
- Visual feedback for analysis tasks
226+
- Cell-by-cell execution for debugging
227+
228+
## Important Files
229+
230+
- **`all_modules.json`**: 112k+ line JSON database of all modules across packages (query this instead of SQLite)
231+
- **`data/stub-packages.json`**: PyPI package metadata
232+
- **`pyproject.toml`**: Dependencies, tool configs (black, ruff, pytest)
233+
- **`pyrightconfig.json`**: Pyright settings for this repo
234+
235+
## Common Pitfalls
236+
237+
1. **Don't edit stubs directly in `stubs/`** - they're generated. Edit sources (docs, frozen, core) or fix generation logic in stubber.
238+
239+
2. **Version format matters**: Use `v1_24_1` (flat) in paths/packages, `v1.24.1` for stubber commands.
240+
241+
3. **Multiple stub packages conflict**: Installing multiple versions overwrites files. Use `--target typings` for isolation.
242+
243+
4. **MCU stubs need hardware**: Generating new MCU stubs requires physical board + firmware flash.
244+
245+
5. **Database queries**: Prefer Data Store MCP tools or JSON files over SQLite CLI (not installed).
246+
247+
## Documentation
248+
249+
- **Read the Docs**: https://micropython-stubs.readthedocs.io/
250+
- **Board Explorer**: https://josverl.github.io/micropython-stubs/board-explorer-mpy.html
251+
- **Key docs**:
252+
- `docs/10_introduction.md` - Stub purpose and benefits
253+
- `docs/stub_types.md` - Stub generation methods
254+
- `docs/diy_stubs_files.md` - Creating custom stubs
255+
256+
## IDE Integration
257+
258+
Stubs are PEP 561 stub-only packages installed via pip:
259+
```bash
260+
pip install micropython-rp2-stubs --target ./typings --no-user
261+
```
262+
263+
Configure IDE to use `./typings` as extra paths for type checking (see `docs/22_vscode.md`, `docs/24_pycharm.md`).
264+
265+
## Publishing
266+
267+
Packages published to PyPI follow naming: `micropython-{port}-stubs` or `micropython-{port}-{board}-stubs`.
268+
269+
Version scheme: `{mpy_version}.post{N}` (e.g., `1.24.1.post1` for first stub update of MicroPython 1.24.1).

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ micropython-stubber
1111
#no secrets
1212
.secrets*
1313

14+
# no .*
15+
.playwright-mcp
16+
1417

1518
# no documentation artefacts
1619
*/~*.pptx
@@ -57,4 +60,4 @@ publish/micropython-stdlib-stubs_100b1
5760

5861
# no test lock files
5962
**/*_lock.file
60-
tests/doc_test/micropython/
63+
tests/doc_test/micropython/

0 commit comments

Comments
 (0)