Skip to content

Commit edd88a5

Browse files
chesterxgchenchesterxgchen
committed
Add notebook testing support and developer testing guide
Features: - Add notebook testing to runtest.sh (-n option) using nbmake - Add transparent dependency caching (auto-expires after 7 days or 50 runs) - Add verbose mode (-v) for detailed pytest output - Add --fresh-deps to force dependency reinstall - Print exact command being executed for debugging Notebook Testing: - Default kernel detection (prefers python3 if available) - Configurable timeout, kernel, and output cleaning - Cell skipping via tags (skip-execution, skip, colab) - Automatic notebook backup and restore during tests Documentation: - New developer_testing.rst: General runtest.sh usage guide - New notebook_testing.rst: Notebook-specific testing guide - Note: Not all notebooks are ready for automated testing yet Usage: ./runtest.sh # Full suite (license, style, unit tests) ./runtest.sh -u # Unit tests only (faster iteration) ./runtest.sh -n # Notebook tests ./runtest.sh -n -v # Notebook tests with verbose output ./runtest.sh --fresh-deps # Force dependency reinstall
1 parent c49eab1 commit edd88a5

File tree

4 files changed

+530
-3
lines changed

4 files changed

+530
-3
lines changed

‎docs/programming_guide.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ Please refer to :ref:`application` for more details.
6666
programming_guide/file_streaming.rst
6767
programming_guide/decomposer_for_large_object
6868
programming_guide/dashboard_api
69+
programming_guide/developer_testing
70+
programming_guide/notebook_testing
6971
user_guide/researcher_guide/index
7072
user_guide/nvflare_cli/nvflare_cli
7173
user_guide/confidential_computing/index

‎docs/programming_guide/developer_testing.rst‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
.. _developer_testing:
2+
3+
##################
4+
Developer Testing
5+
##################
6+
7+
This guide covers the ``runtest.sh`` script used for running tests during NVIDIA FLARE development.
8+
9+
.. contents:: Table of Contents
10+
:local:
11+
:depth: 2
12+
13+
Quick Start
14+
===========
15+
16+
Run the full test suite:
17+
18+
.. code:: bash
19+
20+
./runtest.sh
21+
22+
**Default behavior** (no arguments): Runs the complete CI-equivalent test suite:
23+
24+
1. License header check
25+
2. Code style check (black, isort, flake8)
26+
3. Auto-fix formatting
27+
4. Unit tests with coverage reporting
28+
29+
Individual Commands
30+
-------------------
31+
32+
Run specific tests individually:
33+
34+
.. code:: bash
35+
36+
./runtest.sh -u # Unit tests only (faster, no style checks)
37+
./runtest.sh -s # Check code formatting only
38+
./runtest.sh -f # Fix code formatting only
39+
./runtest.sh -n # Notebook tests
40+
./runtest.sh -l # License header check only
41+
42+
.. note::
43+
44+
``./runtest.sh`` runs the **full suite** (license + style + unit tests).
45+
Use ``./runtest.sh -u`` for **faster iteration** when you only need unit tests.
46+
47+
Available Commands
48+
==================
49+
50+
.. list-table::
51+
:widths: 20 80
52+
:header-rows: 1
53+
54+
* - Option
55+
- Description
56+
* - ``-u`` / ``--unit-tests``
57+
- Run unit tests with pytest
58+
* - ``-s`` / ``--check-format``
59+
- Check code formatting (black, isort, flake8)
60+
* - ``-f`` / ``--fix-format``
61+
- Auto-fix code formatting issues
62+
* - ``-n`` / ``--notebook``
63+
- Run notebook tests using nbmake (see :ref:`notebook_testing`)
64+
* - ``-l`` / ``--check-license``
65+
- Check license headers in source files
66+
* - ``-c`` / ``--coverage``
67+
- Enable coverage reporting (use with ``-u``)
68+
* - ``-r`` / ``--test-report``
69+
- Generate JUnit XML test report (use with ``-u``)
70+
* - ``--clean``
71+
- Clean build artifacts and caches
72+
73+
Common Options
74+
==============
75+
76+
These options can be used with any test command:
77+
78+
.. list-table::
79+
:widths: 25 15 60
80+
:header-rows: 1
81+
82+
* - Option
83+
- Default
84+
- Description
85+
* - ``-v`` / ``--verbose``
86+
- off
87+
- Enable verbose output
88+
* - ``-d`` / ``--dry-run``
89+
- off
90+
- Print commands without executing
91+
* - ``--fresh-deps``
92+
- off
93+
- Force fresh dependency installation (bypass cache)
94+
95+
Dependency Caching
96+
==================
97+
98+
The ``runtest.sh`` script automatically caches dependency installation to speed up repeated runs:
99+
100+
- **First run**: Dependencies are installed and a cache marker is created
101+
- **Subsequent runs**: Dependencies are skipped (uses cache)
102+
- **Auto-refresh**: Cache expires after **7 days** or **50 runs**, whichever comes first
103+
104+
The cache status is displayed at the start of each run:
105+
106+
.. code:: text
107+
108+
Dependencies cached (5/50 runs, 2/7 days) - skipping install
109+
110+
To force a fresh dependency install:
111+
112+
.. code:: bash
113+
114+
# Force reinstall dependencies
115+
./runtest.sh -u --fresh-deps
116+
117+
# Or clear all caches (including dependency cache)
118+
./runtest.sh --clean
119+
120+
Examples
121+
========
122+
123+
Unit Tests
124+
----------
125+
126+
.. code:: bash
127+
128+
# Run all unit tests
129+
./runtest.sh -u
130+
131+
# Run specific test file or directory
132+
./runtest.sh -u tests/unit_test/fuel/
133+
134+
# Run with coverage report
135+
./runtest.sh -u -c
136+
137+
# Run with verbose output
138+
./runtest.sh -u -v
139+
140+
Code Quality
141+
------------
142+
143+
.. code:: bash
144+
145+
# Check formatting (doesn't modify files)
146+
./runtest.sh -s
147+
148+
# Auto-fix formatting issues
149+
./runtest.sh -f
150+
151+
# Check specific directory
152+
./runtest.sh -s nvflare/apis/
153+
154+
Notebook Tests
155+
--------------
156+
157+
See :ref:`notebook_testing` for detailed notebook testing options.
158+
159+
.. code:: bash
160+
161+
# Test default notebook
162+
./runtest.sh -n
163+
164+
# Test specific notebook with verbose output
165+
./runtest.sh -n -v examples/tutorials/flare_simulator.ipynb
166+
167+
Troubleshooting
168+
===============
169+
170+
Slow Repeated Runs
171+
------------------
172+
173+
If runs are slower than expected, check if dependency caching is working:
174+
175+
.. code:: text
176+
177+
Dependencies cached (X/50 runs, Y/7 days) - skipping install
178+
179+
If you see "Dependencies not yet installed", the cache may have expired. This is normal behavior.
180+
181+
Force Clean State
182+
-----------------
183+
184+
To start fresh:
185+
186+
.. code:: bash
187+
188+
./runtest.sh --clean
189+
./runtest.sh -u # Will reinstall dependencies
190+
191+

‎docs/programming_guide/notebook_testing.rst‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
.. _notebook_testing:
2+
3+
#################
4+
Notebook Testing
5+
#################
6+
7+
NVIDIA FLARE uses `nbmake <https://github.com/treebeardtech/nbmake>`__ to test Jupyter notebooks.
8+
This ensures that example notebooks remain functional as the codebase evolves.
9+
10+
.. note::
11+
12+
**Not all notebooks are ready for automated testing yet.** Some notebooks require external
13+
infrastructure (running FLARE servers, provisioned environments, specific datasets) or contain
14+
interactive elements that cannot run in CI. Notebook test coverage is being improved over time.
15+
16+
For general ``runtest.sh`` usage (dependency caching, verbose mode, etc.), see :ref:`developer_testing`.
17+
18+
.. contents:: Table of Contents
19+
:local:
20+
:depth: 2
21+
22+
Quick Start
23+
===========
24+
25+
Use the ``runtest.sh`` script to run notebook tests:
26+
27+
.. code:: bash
28+
29+
# Test default notebook (flare_simulator.ipynb)
30+
./runtest.sh -n
31+
32+
# Test a specific notebook
33+
./runtest.sh -n examples/tutorials/flare_simulator.ipynb
34+
35+
# Test with verbose output
36+
./runtest.sh -n -v examples/tutorials/flare_simulator.ipynb
37+
38+
Notebook-Specific Options
39+
=========================
40+
41+
These options are specific to notebook testing (``-n``):
42+
43+
.. list-table::
44+
:widths: 25 15 60
45+
:header-rows: 1
46+
47+
* - Argument
48+
- Default
49+
- Description
50+
* - ``--timeout=SECONDS``
51+
- 1200
52+
- Timeout in seconds for each notebook execution
53+
* - ``--nb-clean=MODE``
54+
- on-success
55+
- When to clean outputs: ``always``, ``on-success``, ``never``
56+
* - ``--kernel=NAME``
57+
- python3
58+
- Jupyter kernel name (defaults to ``python3`` if available)
59+
60+
Examples
61+
--------
62+
63+
.. code:: bash
64+
65+
# Set a shorter timeout (5 minutes)
66+
./runtest.sh -n --timeout=300 examples/tutorials/flare_simulator.ipynb
67+
68+
# Use a specific kernel
69+
./runtest.sh -n --kernel=python3 examples/tutorials/flare_simulator.ipynb
70+
71+
# Always clean outputs regardless of pass/fail
72+
./runtest.sh -n --nb-clean=always examples/tutorials/
73+
74+
# Combine multiple options with verbose output
75+
./runtest.sh -n -v --timeout=1800 --kernel=python3 examples/tutorials/
76+
77+
Direct pytest Usage
78+
===================
79+
80+
You can also run nbmake directly with pytest:
81+
82+
.. code:: bash
83+
84+
pytest --nbmake --nbmake-timeout=1200 --nbmake-clean=on-success examples/tutorials/
85+
86+
# With specific kernel
87+
pytest --nbmake --nbmake-timeout=1200 --kernel=python3 examples/tutorials/
88+
89+
Skipping Cells in Notebooks
90+
===========================
91+
92+
To skip specific cells during automated testing (e.g., Colab setup cells, interactive
93+
visualizations, or cells that require user input), add one of these tags to the cell metadata:
94+
95+
- ``skip-execution``
96+
- ``skip``
97+
- ``colab``
98+
99+
Adding Tags in Jupyter
100+
----------------------
101+
102+
**In Jupyter Lab:**
103+
104+
1. Select the cell you want to skip
105+
2. Click the gear icon in the right sidebar (or View → Right Sidebar → Show Property Inspector)
106+
3. Under "Common Tools" → "Cell Tags", add: ``skip-execution``
107+
108+
**In Jupyter Notebook (classic):**
109+
110+
1. Select the cell
111+
2. View → Cell Toolbar → Tags
112+
3. Add tag: ``skip-execution``
113+
114+
**In VS Code:**
115+
116+
1. Click on the cell
117+
2. Click "..." menu on the cell
118+
3. Select "Add Cell Tag"
119+
4. Enter: ``skip-execution``
120+
121+
How It Works
122+
============
123+
124+
The testing framework (implemented in ``conftest.py``) automatically:
125+
126+
1. **Before test**: Creates a ``.backup`` of the original notebook
127+
2. **Filters cells**: Removes cells tagged with ``skip-execution``, ``skip``, or ``colab``
128+
3. **Updates kernel**: Adjusts kernel spec to match the specified or detected kernel
129+
4. **Executes**: nbmake runs the notebook through the Jupyter kernel
130+
5. **Restores**: Original notebook is restored from backup
131+
6. **Cleans outputs**: Cell outputs are cleared based on ``--nbmake-clean`` setting
132+
133+
This ensures:
134+
135+
- Original notebooks in git remain unchanged
136+
- Notebooks can contain Colab-specific or interactive cells that won't break CI
137+
- Consistent kernel usage across different development environments
138+
139+
Troubleshooting
140+
===============
141+
142+
Kernel Not Found
143+
----------------
144+
145+
If you see "Kernel not found" errors:
146+
147+
1. Ensure your virtual environment is activated
148+
2. Install ipykernel: ``pip install ipykernel``
149+
3. Register your kernel: ``python -m ipykernel install --user --name=my_env``
150+
4. Or specify an existing kernel: ``./runtest.sh -n --kernel=python3``
151+
152+
Timeout Errors
153+
--------------
154+
155+
For long-running notebooks, increase the timeout:
156+
157+
.. code:: bash
158+
159+
./runtest.sh -n --timeout=3600 examples/advanced/
160+
161+
Notebook Requires External Infrastructure
162+
-----------------------------------------
163+
164+
Some notebooks (e.g., ``flare_api.ipynb``) require a running FLARE server or provisioned
165+
environment. These notebooks will fail in automated testing unless the infrastructure is set up.
166+
167+
For such notebooks, consider:
168+
169+
1. Running them manually in an interactive environment
170+
2. Adding ``skip-execution`` tags to cells that require external services
171+
3. Creating a simplified version for automated testing
172+
173+
Best Practices
174+
==============
175+
176+
1. **Tag cells appropriately**: Mark Colab setup, interactive widgets, and user-input cells with ``skip-execution``
177+
178+
2. **Keep notebooks focused**: Smaller notebooks are faster to test and easier to debug
179+
180+
3. **Use reasonable timeouts**: Increase ``--timeout`` based on expected execution time plus buffer
181+
182+
4. **Test locally before pushing**: Run ``./runtest.sh -n`` on your notebooks before committing
183+
184+
5. **Clean outputs before committing**: Use ``--nb-clean=always`` or manually clear outputs to keep git diffs clean
185+
186+
6. **Use self-contained examples**: Notebooks that use the simulator (like ``flare_simulator.ipynb``) are easier to test than those requiring external servers
187+

0 commit comments

Comments
 (0)