[feat] Add tests (#8)

* write initial tests

* update sample data to use strings for `id` column

* update type hint

* extract fixtures into conftest

* add tests for scoring script

* extract fixtures into conftest; add docstrings

* update python versions to test for

* update readme to mention unit tests

* fix typo

Author: vpchung

.github/workflows/run-tests.yml

@@ -0,0 +1,37 @@
+name: Run tests
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  workflow_dispatch:
+
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+  
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Test with pytest
+        run: |
+          pip install pytest pytest-cov
+          pytest --cov=. --cov-report=xml tests/test_validate.py
+          pytest --cov=. --cov-report=xml tests/test_score.py

README.md

@@ -66,8 +66,9 @@ in Python. R support TBD.
 > for each task.
 
 > [!IMPORTANT]
-> Modifying the `main()` function is highly discouraged. This function has
-> specifically been written to interact with ORCA.
+> The `main()` function is specifically designed for seamless interaction with
+> ORCA. **Modifying this function is strongly discouraged** and could lead to
+> compatibility issues. Unit tests are in place to ensure its proper integration.
 
 3. Update `requirements.txt` with any additional libraries/packages used by the
    script.
@@ -117,8 +118,7 @@ in Python. R support TBD.
 > for each task.
 
 > [!IMPORTANT]
-> Modifying the `main()` function is highly discouraged. This function has
-> specifically been written to interact with ORCA.
+> Like `validate.py`, modifying `main()` is strongly discouraged.
 
 3. Update `requirements.txt` with any additional libraries/packages used by the script.
 

sample_data/groundtruth/truth_data.csv

@@ -1,6 +1,6 @@
 id,disease
-1,1
-2,1
-3,0
-4,1
-5,1
+A_01,1
+A_02,1
+A_03,0
+A_04,1
+A_05,1

sample_data/invalid_pred.csv

@@ -1,7 +1,7 @@
 id,probability,note
-1,-0.011,invalid value (bad range)
-2,NA,invalid value (null)
-3,0.092,
-3,0.913,invalid ID (duplicate)
-5,0.543,
-10,0.290,invalid ID (unkonwn)
+A_01,-0.011,invalid value (bad range)
+A_02,NA,invalid value (null)
+A_03,0.092,
+A_03,0.913,invalid ID (duplicate)
+A_05,0.543,
+A_10,0.290,invalid ID (unkonwn)

sample_data/valid_pred.csv

@@ -1,6 +1,6 @@
 id,probability
-1,0.011
-2,0.765
-3,0.092
-4,0.913
-5,0.543
+A_01,0.011
+A_02,0.765
+A_03,0.092
+A_04,0.913
+A_05,0.543

tests/conftest.py

@@ -0,0 +1,78 @@
+import json
+import os
+import sys
+import tempfile
+
+import pandas as pd
+import pytest
+
+
+def pytest_configure(config):
+    """Allow test scripts to import scripts from parent folder."""
+    src_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+    sys.path.insert(0, src_path)
+
+
+@pytest.fixture(scope="module")
+def temp_dir():
+    """Creates a temporary directory for test files."""
+    with tempfile.TemporaryDirectory() as tmpdir:
+        yield tmpdir
+
+
+@pytest.fixture(scope="module")
+def groundtruth_dir(temp_dir):
+    """Creates a temporary groundtruth directory."""
+    gt_dir = os.path.join(temp_dir, "groundtruth")
+    os.makedirs(gt_dir)
+    return gt_dir
+
+
+@pytest.fixture(scope="module")
+def gt_file(groundtruth_dir):
+    """Creates a dummy groundtruth file."""
+    truth = pd.DataFrame(
+        {
+            "id": ["A_01", "A_02", "A_03"],
+            "disease": [1, 0, 1],
+        }
+    )
+    gt_file = os.path.join(groundtruth_dir, "truth.csv")
+    truth.to_csv(gt_file, index=False)
+    return gt_file
+
+
+@pytest.fixture(scope="module")
+def pred_file(temp_dir):
+    """Creates a dummy prediction file."""
+    pred = pd.DataFrame(
+        {
+            "id": ["A_01", "A_02", "A_03"],
+            "probability": [0.5, 0.5, 0.5],
+        }
+    )
+    pred_file = os.path.join(temp_dir, "predictions.csv")
+    pred.to_csv(pred_file, index=False)
+    return pred_file
+
+
+@pytest.fixture(scope="module")
+def valid_predictions_json():
+    """Creates a dummy valid results JSON."""
+    return json.dumps(
+        {
+            "validation_status": "VALIDATED",
+            "validation_errors": "",
+        }
+    )
+
+
+@pytest.fixture(scope="module")
+def invalid_predictions_json():
+    """Creates a dummy invalid results JSON."""
+    return json.dumps(
+        {
+            "validation_status": "INVALID",
+            "validation_errors": "Something went wrong.",
+        }
+    )