Skip to content

Commit d351909

Browse files
albanpuechalbanpuech
authored
Merge pull request #54 from gridfm/opf_integration_with_dc_results
Opf integration with dc results
2 parents 11a75f8 + b016e7e commit d351909

67 files changed

Lines changed: 1291 additions & 2081 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 93 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,9 @@ gridfm_graphkit <command> [OPTIONS]
6060
Available commands:
6161

6262
* `train` - Train a new model from scratch
63-
* `finetune` – Fine-tune an existing pre-trained model
64-
* `evaluate` – Evaluate model performance on a dataset
65-
* `predict` – Run inference and save predictions
63+
* `finetune` - Fine-tune an existing pre-trained model
64+
* `evaluate` - Evaluate model performance on a dataset
65+
* `predict` - Run inference and save predictions
6666

6767
---
6868

@@ -74,13 +74,22 @@ gridfm_graphkit train --config path/to/config.yaml
7474

7575
### Arguments
7676

77-
| Argument | Type | Description | Default |
78-
| ---------------- | ------ | ---------------------------------------------------------------- | ------- |
79-
| `--config` | `str` | **Required**. Path to the training configuration YAML file. | `None` |
80-
| `--exp_name` | `str` | MLflow experiment name. | `timestamp` |
81-
| `--run_name` | `str` | MLflow run name. | `run` |
82-
| `--log_dir` | `str` | MLflow tracking/logging directory. | `mlruns` |
83-
| `--data_path` | `str` | Root dataset directory. | `data` |
77+
| Argument | Type | Description | Default |
78+
| -------- | ---- | ----------- | ------- |
79+
| `--config` | `str` | **Required**. Path to the training configuration YAML file. | `None` |
80+
| `--exp_name` | `str` | MLflow experiment name. | `timestamp` |
81+
| `--run_name` | `str` | MLflow run name. | `run` |
82+
| `--log_dir` | `str` | MLflow tracking/logging directory. | `mlruns` |
83+
| `--data_path` | `str` | Root dataset directory. | `data` |
84+
| `--compile [MODE]` | `str` | Enable `torch.compile` mode. Valid values: `default`, `reduce-overhead`, `max-autotune`, `max-autotune-no-cudagraphs`. If flag is passed without a value, mode is `default`. | `None` |
85+
| `--bfloat16` | `flag` | Cast model to `torch.bfloat16` (`model.to(torch.bfloat16)`). | `False` |
86+
| `--tf32` | `flag` | Enable TF32 on Ampere+ GPUs via `torch.set_float32_matmul_precision("high")`. | `False` |
87+
| `--dataset_wrapper` | `str` | Registered dataset wrapper name (see `DATASET_WRAPPER_REGISTRY`), e.g. `SharedMemoryCacheDataset`. | `None` |
88+
| `--plugins` | `list[str]` | Python packages to import for plugin registration, e.g. `gridfm_graphkit_ee`. | `[]` |
89+
| `--num_workers` | `int` | Override `data.workers` from YAML. Use `0` to debug worker crashes. | `None` |
90+
| `--dataset_wrapper_cache_dir` | `str` | Disk cache directory for dataset wrapper; cache is loaded from here when present and saved after first population. | `None` |
91+
| `--profiler` | `str` | Enable Lightning profiler (`simple`, `advanced`, `pytorch`). | `None` |
92+
| `--compute_dc_ac_metrics` | `flag` | Compute ground-truth AC/DC power balance metrics on the test split. | `False` |
8493

8594
### Examples
8695

@@ -100,14 +109,23 @@ gridfm_graphkit finetune --config path/to/config.yaml --model_path path/to/model
100109

101110
### Arguments
102111

103-
| Argument | Type | Description | Default |
104-
| -------------- | ----- | ----------------------------------------------- | --------- |
105-
| `--config` | `str` | **Required**. Fine-tuning configuration file. | `None` |
106-
| `--model_path` | `str` | **Required**. Path to a pre-trained model state dict. | `None` |
107-
| `--exp_name` | `str` | MLflow experiment name. | timestamp |
108-
| `--run_name` | `str` | MLflow run name. | `run` |
109-
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
110-
| `--data_path` | `str` | Root dataset directory. | `data` |
112+
| Argument | Type | Description | Default |
113+
| -------- | ---- | ----------- | ------- |
114+
| `--config` | `str` | **Required**. Fine-tuning configuration file. | `None` |
115+
| `--model_path` | `str` | **Required**. Path to a pre-trained model state dict. | `None` |
116+
| `--exp_name` | `str` | MLflow experiment name. | `timestamp` |
117+
| `--run_name` | `str` | MLflow run name. | `run` |
118+
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
119+
| `--data_path` | `str` | Root dataset directory. | `data` |
120+
| `--compile [MODE]` | `str` | Enable `torch.compile` mode. Valid values: `default`, `reduce-overhead`, `max-autotune`, `max-autotune-no-cudagraphs`. If flag is passed without a value, mode is `default`. | `None` |
121+
| `--bfloat16` | `flag` | Cast model to `torch.bfloat16` (`model.to(torch.bfloat16)`). | `False` |
122+
| `--tf32` | `flag` | Enable TF32 on Ampere+ GPUs via `torch.set_float32_matmul_precision("high")`. | `False` |
123+
| `--dataset_wrapper` | `str` | Registered dataset wrapper name (see `DATASET_WRAPPER_REGISTRY`), e.g. `SharedMemoryCacheDataset`. | `None` |
124+
| `--plugins` | `list[str]` | Python packages to import for plugin registration, e.g. `gridfm_graphkit_ee`. | `[]` |
125+
| `--num_workers` | `int` | Override `data.workers` from YAML. Use `0` to debug worker crashes. | `None` |
126+
| `--dataset_wrapper_cache_dir` | `str` | Disk cache directory for dataset wrapper; cache is loaded from here when present and saved after first population. | `None` |
127+
| `--profiler` | `str` | Enable Lightning profiler (`simple`, `advanced`, `pytorch`). | `None` |
128+
| `--compute_dc_ac_metrics` | `flag` | Compute ground-truth AC/DC power balance metrics on the test split. | `False` |
111129

112130

113131
---
@@ -120,17 +138,25 @@ gridfm_graphkit evaluate --config path/to/eval.yaml --model_path path/to/model.p
120138

121139
### Arguments
122140

123-
| Argument | Type | Description | Default |
124-
| --------------------- | ----- | ------------------------------------------------------------------------------------------------------------- | --------- |
125-
| `--config` | `str` | **Required**. Path to evaluation config. | `None` |
126-
| `--model_path` | `str` | Path to the trained model state dict. | `None` |
127-
| `--normalizer_stats` | `str` | Path to `normalizer_stats.pt` from a training run. Restores `fit_on_train` normalizers from saved statistics instead of re-fitting on the current data split. | `None` |
128-
| `--exp_name` | `str` | MLflow experiment name. | timestamp |
129-
| `--run_name` | `str` | MLflow run name. | `run` |
130-
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
131-
| `--data_path` | `str` | Dataset directory. | `data` |
132-
| `--compute_dc_ac_metrics` | `flag` | Compute ground-truth AC/DC power balance metrics on the test split. | `False` |
133-
| `--save_output` | `flag` | Save predictions as `<grid_name>_predictions.parquet` under MLflow artifacts (`.../artifacts/test`). | `False` |
141+
| Argument | Type | Description | Default |
142+
| -------- | ---- | ----------- | ------- |
143+
| `--config` | `str` | **Required**. Path to evaluation config. | `None` |
144+
| `--model_path` | `str` | Path to the trained model state dict. | `None` |
145+
| `--normalizer_stats` | `str` | Path to `normalizer_stats.pt` from a training run. Restores `fit_on_train` normalizers from saved statistics instead of re-fitting on current split. | `None` |
146+
| `--exp_name` | `str` | MLflow experiment name. | `timestamp` |
147+
| `--run_name` | `str` | MLflow run name. | `run` |
148+
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
149+
| `--data_path` | `str` | Dataset directory. | `data` |
150+
| `--compile [MODE]` | `str` | Enable `torch.compile` mode. Valid values: `default`, `reduce-overhead`, `max-autotune`, `max-autotune-no-cudagraphs`. If flag is passed without a value, mode is `default`. | `None` |
151+
| `--bfloat16` | `flag` | Cast model to `torch.bfloat16` (`model.to(torch.bfloat16)`). | `False` |
152+
| `--tf32` | `flag` | Enable TF32 on Ampere+ GPUs via `torch.set_float32_matmul_precision("high")`. | `False` |
153+
| `--dataset_wrapper` | `str` | Registered dataset wrapper name (see `DATASET_WRAPPER_REGISTRY`), e.g. `SharedMemoryCacheDataset`. | `None` |
154+
| `--plugins` | `list[str]` | Python packages to import for plugin registration, e.g. `gridfm_graphkit_ee`. | `[]` |
155+
| `--num_workers` | `int` | Override `data.workers` from YAML. Use `0` to debug worker crashes. | `None` |
156+
| `--dataset_wrapper_cache_dir` | `str` | Disk cache directory for dataset wrapper; cache is loaded from here when present and saved after first population. | `None` |
157+
| `--profiler` | `str` | Enable Lightning profiler (`simple`, `advanced`, `pytorch`). | `None` |
158+
| `--compute_dc_ac_metrics` | `flag` | Compute ground-truth AC/DC power balance metrics on the test split. | `False` |
159+
| `--save_output` | `flag` | Save predictions as `<grid_name>_predictions.parquet` under MLflow artifacts (`.../artifacts/test`). | `False` |
134160

135161
### Example with saved normalizer stats
136162

@@ -156,16 +182,44 @@ gridfm_graphkit predict --config path/to/config.yaml --model_path path/to/model.
156182

157183
### Arguments
158184

159-
| Argument | Type | Description | Default |
160-
| --------------------- | ----- | ------------------------------------------------------------------------------------------------------------- | --------- |
161-
| `--config` | `str` | **Required**. Path to prediction config file. | `None` |
162-
| `--model_path` | `str` | Path to the trained model state dict. | `None` |
163-
| `--normalizer_stats` | `str` | Path to `normalizer_stats.pt` from a training run. Restores `fit_on_train` normalizers from saved statistics. | `None` |
164-
| `--exp_name` | `str` | MLflow experiment name. | timestamp |
165-
| `--run_name` | `str` | MLflow run name. | `run` |
166-
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
167-
| `--data_path` | `str` | Dataset directory. | `data` |
168-
| `--output_path` | `str` | Directory where predictions are saved as `<grid_name>_predictions.parquet`. | `data` |
185+
| Argument | Type | Description | Default |
186+
| -------- | ---- | ----------- | ------- |
187+
| `--config` | `str` | **Required**. Path to prediction config file. | `None` |
188+
| `--model_path` | `str` | Path to trained model state dict. Optional; may be defined in config. | `None` |
189+
| `--normalizer_stats` | `str` | Path to `normalizer_stats.pt` from a training run. Restores `fit_on_train` normalizers from saved statistics. | `None` |
190+
| `--exp_name` | `str` | MLflow experiment name. | `timestamp` |
191+
| `--run_name` | `str` | MLflow run name. | `run` |
192+
| `--log_dir` | `str` | MLflow logging directory. | `mlruns` |
193+
| `--data_path` | `str` | Dataset directory. | `data` |
194+
| `--dataset_wrapper` | `str` | Registered dataset wrapper name (see `DATASET_WRAPPER_REGISTRY`), e.g. `SharedMemoryCacheDataset`. | `None` |
195+
| `--plugins` | `list[str]` | Python packages to import for plugin registration, e.g. `gridfm_graphkit_ee`. | `[]` |
196+
| `--num_workers` | `int` | Override `data.workers` from YAML. Use `0` to debug worker crashes. | `None` |
197+
| `--dataset_wrapper_cache_dir` | `str` | Disk cache directory for dataset wrapper; cache is loaded from here when present and saved after first population. | `None` |
198+
| `--output_path` | `str` | Directory where predictions are saved as `<grid_name>_predictions.parquet`. | `data` |
199+
| `--compile [MODE]` | `str` | Enable `torch.compile` mode. Valid values: `default`, `reduce-overhead`, `max-autotune`, `max-autotune-no-cudagraphs`. If flag is passed without a value, mode is `default`. | `None` |
200+
| `--bfloat16` | `flag` | Cast model to `torch.bfloat16` (`model.to(torch.bfloat16)`). | `False` |
201+
| `--tf32` | `flag` | Enable TF32 on Ampere+ GPUs via `torch.set_float32_matmul_precision("high")`. | `False` |
202+
| `--profiler` | `str` | Enable Lightning profiler (`simple`, `advanced`, `pytorch`). | `None` |
203+
204+
---
205+
206+
## Benchmarking Dataloader Throughput
207+
208+
```bash
209+
gridfm_graphkit benchmark --config path/to/config.yaml
210+
```
211+
212+
### Arguments
213+
214+
| Argument | Type | Description | Default |
215+
| -------- | ---- | ----------- | ------- |
216+
| `--config` | `str` | **Required**. Path to configuration YAML file. | `None` |
217+
| `--data_path` | `str` | Root dataset directory. | `data` |
218+
| `--epochs` | `int` | Number of epochs to iterate through the train dataloader. | `3` |
219+
| `--dataset_wrapper` | `str` | Registered dataset wrapper name (see `DATASET_WRAPPER_REGISTRY`), e.g. `SharedMemoryCacheDataset`. | `None` |
220+
| `--dataset_wrapper_cache_dir` | `str` | Directory for dataset wrapper disk cache. | `None` |
221+
| `--num_workers` | `int` | Override `data.workers` from YAML. | `None` |
222+
| `--plugins` | `list[str]` | Python packages to import for plugin registration. | `[]` |
169223

170224
Use built-in help for full command details:
171225

‎docs/datasets/data_modules.md‎

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎docs/datasets/data_normalization.md‎

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎docs/datasets/powergrid.md‎

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎docs/datasets/transforms.md‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎docs/install/installation.md‎

Lines changed: 14 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,32 @@
11
# Installation
22

3-
You can install `gridfm-graphkit` directly from PyPI:
3+
The steps below mirror the [README](https://github.com/gridfm/gridfm-graphkit/blob/main/README.md#installation). Run them from the root of a local clone or source checkout of the repository.
44

5-
```bash
6-
pip install gridfm-graphkit
7-
```
8-
9-
For GPU support and compatibility with PyTorch Geometric's scatter operations, install PyTorch (and optionally CUDA) first, then install the matching `torch-scatter` wheel. See [PyTorch and torch-scatter](#pytorch-and-torch-scatter-optional) below.
10-
11-
---
12-
13-
## Development Setup
14-
15-
To contribute or develop locally, clone the repository and install in editable mode. Use Python 3.10, 3.11, or 3.12 (3.12 is recommended).
5+
Create and activate a virtual environment (make sure you use the right python version = 3.10, 3.11 or 3.12. I highly recommend 3.12)
166

177
```bash
18-
git clone git@github.com:gridfm/gridfm-graphkit.git
19-
cd gridfm-graphkit
208
python -m venv venv
219
source venv/bin/activate
22-
pip install -e .
2310
```
2411

25-
### PyTorch and torch-scatter (optional)
12+
Install gridfm-graphkit in editable mode
2613

27-
If you need GPU acceleration or PyTorch Geometric scatter ops (used by the library), install PyTorch and the matching `torch-scatter` wheel:
14+
```bash
15+
pip install -e .
16+
```
2817

29-
1. Install PyTorch (see [pytorch.org](https://pytorch.org/) for your platform and CUDA version).
18+
Get PyTorch + CUDA version for torch-scatter
3019

31-
2. Get your Torch + CUDA version string:
32-
```bash
33-
TORCH_CUDA_VERSION=$(python -c "import torch; print(torch.__version__ + ('+cpu' if torch.version.cuda is None else ''))")
34-
```
20+
```bash
21+
TORCH_CUDA_VERSION=$(python -c "import torch; print(torch.__version__ + ('+cpu' if torch.version.cuda is None else ''))")
22+
```
3523

36-
3. Install the correct `torch-scatter` wheel:
37-
```bash
38-
pip install torch-scatter -f https://data.pyg.org/whl/torch-${TORCH_CUDA_VERSION}.html
39-
```
24+
Install the correct torch-scatter wheel
4025

41-
---
26+
```bash
27+
pip install torch-scatter -f https://data.pyg.org/whl/torch-${TORCH_CUDA_VERSION}.html
28+
```
4229

43-
## Optional extras
4430

4531
For documentation generation and unit testing, install with the optional `dev` and `test` extras:
4632

‎docs/models/models.md‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

0 commit comments

Comments
 (0)