Docs: Eval readme added

Author: rahulkfernandes

financial_loss_functions/scripts/run_evaluation.py

@@ -83,14 +83,15 @@ def cleanup_on_interrupt():
             '-pm',
             '--prev_mode', 
             choices=['one_model', 'one'], 
-            help='Choose, models to evaluate. This can be for selected models or a single model'
+            default='one_model',
+            help='Choose the previous grid mode used for tuning. This is used to collect artifacts'
         )
 
         parser.add_argument(
             '-m',
             '--model_losses',
             nargs='+',
-            help="Model Loss combination name must be in the format <ModelName>-<LossName>"
+            help='Model Loss combination name must be in the format <ModelName>-<LossName>'
         )
 
         parser.add_argument('-mpi', '--mpi', action='store_true', help='MPI for HPC')

financial_loss_functions/src/evaluation/README.md

@@ -1 +1,72 @@
-# Evaluation on Test Data
+# Evaluation on Test Data
+[<- Back to Main README](/financial_loss_functions/README.md)
+
+## Evaluate Selected Candidates on the Test Data
+
+### Run Evaluation on Test Data
+```bash
+python -m scripts.run_evaluation [--prev_mode PREVIOUS MODE] [OPTIONS]
+```
+
+#### Arguments Reference
+| Flag | Long Flag | Choices / Type | Default | Notes |
+| :--- | :--- | :--- | :--- | :--- |
+| `-pm` | `--prev_mode` | `one_model`, `one` | `one_model` | The grid mode used in the previous tuning stage. |
+| `-m` | `--model_losses` | `[<MODEL_NAME1>-<LOSS_NAME1>]` | None | List of model-loss combinations to run the test evaluation |
+| `-mpi` | `--mpi` | *None* | None | Flag to use mpi for distributed evaluation if grid mode is "one_model" |
+
+#### Examples:
+1. Run a selected collection of model-loss combinations on a HPC cluster.
+```bash
+srun python -m scripts.run_evaluation --model_losses <MODEL_NAME1>-<LOSS_NAME1> <MODEL_NAME2>-<LOSS_NAME2> --mpi
+```
+
+To run this sequentially (not recommended):
+```bash
+python -m scripts.run_evaluation --model_losses <MODEL_NAME1>-<LOSS_NAME1> <MODEL_NAME2>-<LOSS_NAME2>
+```
+
+2. Run specific model-loss combination (no mpi, flag will be ignored):
+```bash
+python -m scripts.run_evaluation --model_losses <MODEL_NAME1>-<LOSS_NAME1>
+```
+
+3. Run specific model-loss combination when the previously used mode was 'one':
+```bash
+python -m scripts.run_evaluation --prev_mode one --model_losses <MODEL_NAME1>-<LOSS_NAME1>
+``
+
+4. To get help
+```bash
+python -m scripts.run_evaluation --help
+```
+
+#### Example SBATCH Script
+```bash
+#!/bin/bash -l
+
+#SBATCH --job-name=<JOB NAME>
+#SBATCH --comment="<DESCRIPTION>"
+
+#SBATCH --account=<ACC. NAME>
+#SBATCH --partition=<PARTITION NAME>		# Partition to run your job on
+
+#SBATCH --output=<OUTPUT FILE PATH>		    # Output file
+#SBATCH --error=<ERROR FILE PATH>			# Error file
+
+#SBATCH --time=0-28:00:00		# Time limit - d:hh:mm:ss
+#SBATCH --nodes=1			    # How many nodes to run on
+#SBATCH --ntasks-per-node=8     # Ranks per node
+#SBATCH --cpus-per-task=2		# Number of CPUs per task
+#SBATCH --mem-per-cpu=4g		# Memory per CPU
+#SBATCH --gres=gpu:2            # GPUs per node
+
+
+# Running Code
+source /home/<USER_NAME>/miniconda3/etc/profile.d/conda.sh
+conda activate <ENV_NAME>
+
+srun python -m scripts.run_evaluation -pm one_model \
+-m <MODEL_NAME1>-<LOSS_NAME1> <MODEL_NAME2>-<LOSS_NAME2> -mpi
+```
+All SBATCH examples can be found at `./scripts/sample_batch_scripts/`. The example above can be found at [sample_batch_script](../../scripts/sample_batch_scripts/sample_test_all.sh).

financial_loss_functions/src/training/README.md

@@ -35,7 +35,7 @@ To run this sequentially (not recommended):
 python -m scripts.run_training --model '<Model Name>' --tune
 ```
 
-2. Run specific model-loss combination with tuning (no mpi):
+2. Run specific model-loss combination with tuning (no mpi, flag will be ignored):
 ```bash
 python -m scripts.run_training --grid_mode one --model '<Model Name>' --loss_name '<Loss Name>' --tune
 ```
@@ -72,7 +72,7 @@ conda activate <ENV_NAME>
 
 srun python -m scripts.run_training -gm one_model -m <MODEL_NAME> -t -mpi
 ```
-All SBATCH examples can be found at `./scripts/sample_batch_scripts/`. The example above can be found at [sample_batch_scripts](../../scripts/sample_batch_scripts/sample_one_model_tune.sh).
+All SBATCH examples can be found at `./scripts/sample_batch_scripts/`. The example above can be found at [sample_batch_script](../../scripts/sample_batch_scripts/sample_one_model_tune.sh).
 
 ## Available Models and Loss Functions
 ### Neural Network Models