Merge branch 'main' into paper/JOSS-submission

Author: Asanto32

README.md

@@ -36,7 +36,7 @@ The main processing pipeline of the wristpy module can be described as follows:
 - ***Data imputation*** In the special case when dealing with the Actigraph `idle_sleep_mode == enabled`, the gaps in acceleration are filled in after calibration, to avoid biasing the calibration phase.
 - **Metrics Calculation**: Calculates various activity metrics on the calibrated data, namely ENMO (Euclidean norm, minus one), MAD (mean amplitude deviation) <sup>1</sup>, Actigraph activity counts<sup>2</sup>, MIMS (monitor-independent movement summary) unit <sup>3</sup>, and angle-Z (angle of acceleration relative to the *x-y* axis).
 - **Non-wear detection**: We find periods of non-wear based on the acceleration data. Specifically, the standard deviation of the acceleration values in a given time window, along each axis, is used as a threshold to decide `wear` or `not wear`. Additionally, we can use the temperature sensor, when avaia\lable, to augment the acceleration data. This is used in the CTA (combined temperature and acceleration) algorithm <sup>4</sup>, and in the `skdh` DETACH algorithm <sup>5</sup>. Furthermore, ensemble classification of non-wear periods is possible by providing a list (of any length) of non-wear algorithm options.
-- **Sleep Detection**: Using the HDCZ<sup>6</sup> and HSPT<sup>7</sup> algorithms to analyze changes in arm angle we are able to find periods of sleep. We find the sleep onset-wakeup times for all sleep windows detected. Any sleep periods that overlap with detected non-wear times are removed, and any remaining sleep periods shorter than 15 minutes (default value) are removed.
+- **Sleep Detection**: Using the HDCZ<sup>6</sup> and HSPT<sup>7</sup> algorithms to analyze changes in arm angle we are able to find periods of sleep. We find the sleep onset-wakeup times for all sleep windows detected. Any sleep periods that overlap with detected non-wear times are removed, and any remaining sleep periods shorter than 15 minutes (default value) are removed. Additionally, the SIB (sustained inactivity bouts) and the SPT (sleep period time) windows are provided as part of the output to aid in sleep metric post-processing.
 - **Physical activity levels**: Using the chosen physical activity metric (aggregated into time bins, 5 second default) we compute activity levels into the following categories: [`inactive`, `light`, `moderate`, `vigorous`]. The threshold values can be defined by the user, while the default values are chosen based on the specific activity metric and the values found in the literature  <sup>8-10</sup>.
 - **Data output**: The output results can be saved in `.csv` or `.parquet` data formats, with the run-time configuration parameters saved in a `.json` dictionary.
 
@@ -102,8 +102,11 @@ results = orchestrator.run(
 physical_activity_metric = results.physical_activity_metric
 anglez = results.anglez
 physical_activity_levels = results.physical_activity_levels
-nonwear_array = results.nonwear_epoch
-sleep_windows = results.sleep_windows_epoch
+nonwear_array = results.nonwear_status
+sleep_windows = results.sleep_status
+sib_periods = results.sib_periods
+spt_periods = results.spt_periods
+
 ```
 #### Running entire directories:
 ```Python
@@ -131,8 +134,11 @@ subject1 = results_dict['subject1']
 physical_activity_metric = subject1.physical_activity_metric
 anglez = subject1.anglez
 physical_activity_levels = subject1.physical_activity_levels
-nonwear_array = subject1.nonwear_epoch
-sleep_windows = subject1.sleep_windows_epoch
+nonwear_array = subject1.nonwear_status
+sleep_windows = subject1.sleep_status
+sib_periods = subject1.sib_periods
+spt_periods = subject1.spt_periods
+
 ```
 
 ### Using Wristpy Through Docker

dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.11-buster
+FROM python:3.11-bookworm
 
 WORKDIR /app
 COPY . /app/

docs/wristpy_tutorial.md

@@ -33,7 +33,7 @@ results = orchestrator.run(
 This runs the processing pipeline with all the default arguments, creates an output `.csv` file, a `.json` file with the pipeline configuration parameters, and will create a `results` object that contains the various output metrics (namely; the specified physical activity metric, angle-z, physical activity classification values, non-wear status, and sleep status).
 
 
-The orchestrator can also process entire directories. The call to the orchestrator remains largely the same but now output is expected to be a directory and the desired filetype for the saved files **must** be specified:
+The orchestrator can also process entire directories. The call to the orchestrator remains largely the same but now output is expected to be a directory and the desired filetype for the saved files can be specified through the output_filetype arguement(default value is ".csv"):
 
 ```python
 from wristpy.core import orchestrator
@@ -45,6 +45,28 @@ results = orchestrator.run(
 )
 ```
 
+If users would prefer to process specific files instead of entire directories we recommend looping through a list of file names. The following code snipet will save results objects into a dictionary, and the output files into the desired directory:
+
+```python
+from wristpy.core import orchestrator
+import pathlib
+
+file_path = pathlib.Path("/path/to/data/")
+output_dir = pathlib.Path("/path/to/save/dir/")
+file_names = [pathlib.Path("file1.gt3x"), pathlib.Path("file2.gt3x"), pathlib.Path("file3.gt3x")]
+results_dict = {}
+
+for file in file_names:
+    input_path = file_path / file
+    output_path = output_dir / file.stem + ".csv"
+    result = orchestrator.run(
+        input = input_path
+        output = output_path
+    )
+    results_dict[file.stem] = result
+```
+
+
 
 
 We can visualize some of the outputs within the `results` object, directly, with the following scripts:
@@ -74,6 +96,8 @@ We can also view and process these outputs from the saved `.csv` output file:
 import polars as pl
 import matplotlib.pyplot as plt
 
+output_results = pl.read_csv('path/to/save/file_name.csv', try_parse_dates=True)
+
 activity_mapping = {
     "inactive": 0,
     "light": 1,

src/wristpy/core/cli.py

@@ -6,7 +6,7 @@
 
 import typer
 
-from wristpy.core import config
+from wristpy.core import config, exceptions
 
 logger = config.get_logger()
 app = typer.Typer(
@@ -74,11 +74,10 @@ def main(
         help="Path where data will be saved. Supports .csv and .parquet formats.",
     ),
     output_filetype: OutputFileType = typer.Option(
-        None,
+        ".csv",
         "-O",
         "--output-filetype",
-        help="Format for save files when processing directories. "
-        "Leave as None when processing single files.",
+        help="Format for save files when processing directories. ",
     ),
     calibrator: Calibrator = typer.Option(
         Calibrator.none,
@@ -150,17 +149,21 @@ def main(
     calibrator_value = None if calibrator == Calibrator.none else calibrator.value
 
     logger.debug("Running wristpy. arguments given: %s", locals())
-    orchestrator.run(
-        input=input,
-        output=output,
-        calibrator=calibrator_value,
-        activity_metric=activity_metric.value,
-        thresholds=None if thresholds is None else thresholds,
-        epoch_length=epoch_length,
-        nonwear_algorithm=nonwear_algorithms,  # type: ignore[arg-type] # Covered by NonwearAlgorithm Enum class
-        verbosity=log_level,
-        output_filetype=output_filetype.value if output_filetype else None,  # type: ignore[arg-type] # Covered by OutputFileType Enum class
-    )
+    try:
+        orchestrator.run(
+            input=input,
+            output=output,
+            calibrator=calibrator_value,
+            activity_metric=activity_metric.value,
+            thresholds=None if thresholds is None else thresholds,
+            epoch_length=epoch_length,
+            nonwear_algorithm=nonwear_algorithms,  # type: ignore[arg-type] # Covered by NonwearAlgorithm Enum class
+            verbosity=log_level,
+            output_filetype=output_filetype.value,
+        )
+    except exceptions.EmptyDirectoryError as e:
+        typer.echo(f"Error: {e}", err=True)
+        raise typer.Exit(1)
 
 
 if __name__ == "__main__":

src/wristpy/core/exceptions.py

@@ -40,3 +40,9 @@ class InvalidFileTypeError(LoggedException):
     """Wristpy did not expect this file extension."""
 
     pass
+
+
+class EmptyDirectoryError(LoggedException):
+    """No .gt3x or .bin files were found in the directory."""
+
+    pass

src/wristpy/core/orchestrator.py

@@ -13,7 +13,7 @@
     calibration,
     idle_sleep_mode_imputation,
     metrics,
-    nonwear_utils,
+    processing_utils,
 )
 
 logger = config.get_logger()
@@ -33,7 +33,7 @@ def run(
     activity_metric: Literal["enmo", "mad", "ag_count", "mims"] = "enmo",
     nonwear_algorithm: Sequence[Literal["ggir", "cta", "detach"]] = ["ggir"],
     verbosity: int = logging.WARNING,
-    output_filetype: Optional[Literal[".csv", ".parquet"]] = None,
+    output_filetype: Literal[".csv", ".parquet"] = ".csv",
 ) -> Union[writers.OrchestratorResults, Dict[str, writers.OrchestratorResults]]:
     """Runs main processing steps for wristpy on single files, or directories.
 
@@ -59,8 +59,8 @@ def run(
         activity_metric: The metric to be used for physical activity categorization.
         nonwear_algorithm: The algorithm to be used for nonwear detection.
         verbosity: The logging level for the logger.
-        output_filetype: Specifies the data format for the save files. Must be None when
-            processing files, must be a valid file type when processing directories.
+        output_filetype: Specifies the data format for the save files. Only used when
+            processing directories.
 
     Returns:
         All calculated data in a save ready format as a Results object or as a
@@ -69,8 +69,7 @@ def run(
     Raises:
         ValueError: If the physical activity thresholds are not unique or not in
             ascending order.
-        ValueError: If processing a file and the output_filetype is not None
-        ValueError: If output is None but output_filetype is not None.
+
 
     References:
         [1] Hildebrand, M., et al. (2014). Age group comparability of raw accelerometer
@@ -105,13 +104,6 @@ def run(
         raise ValueError(message)
 
     if input.is_file():
-        if output_filetype is not None:
-            raise ValueError(
-                "When processing single files, output_filetype should be None - "
-                "the file type will be determined from the output path."
-            )
-        logger.debug("Input is file, forwarding to run_file with output=%s", output)
-
         return _run_file(
             input=input,
             output=output,
@@ -147,14 +139,13 @@ def _run_directory(
     epoch_length: float = 5,
     nonwear_algorithm: Sequence[Literal["ggir", "cta", "detach"]] = ["ggir"],
     verbosity: int = logging.WARNING,
-    output_filetype: Optional[Literal[".csv", ".parquet"]] = None,
+    output_filetype: Literal[".csv", ".parquet"] = ".csv",
     activity_metric: Literal["enmo", "mad", "ag_count", "mims"] = "enmo",
 ) -> Dict[str, writers.OrchestratorResults]:
     """Runs main processing steps for wristpy on  directories.
 
     The run_directory() function will execute the run_file() function on entire
-    directories. The input and output (if any) paths must directories. An
-    output_filetype must be specified if and only if an output is given. Output file
+    directories. The input and output (if any) paths must directories. Output file
     names will be derived from input file names.
 
 
@@ -192,9 +183,6 @@ def _run_directory(
         Activity: Retrospective Observational Data Analysis Study JMIR Mhealth Uhealth
         2022;10(7):e38077 URL: https://mhealth.jmir.org/2022/7/e38077 DOI: 10.2196/38077
     """
-    if output is None and output_filetype is not None:
-        raise ValueError("If no output is given, output_filetype must be None.")
-
     if output is not None:
         if output.is_file():
             raise ValueError(
@@ -209,12 +197,14 @@ def _run_directory(
     file_names = list(itertools.chain(input.glob("*.gt3x"), input.glob("*.bin")))
 
     if not file_names:
-        raise FileNotFoundError(f"Directory {input} contains no .gt3x or .bin files.")
+        raise exceptions.EmptyDirectoryError(
+            f"Directory {input} contains no .gt3x or .bin files."
+        )
 
     results_dict = {}
     for file in file_names:
         output_file_path = (
-            output / pathlib.Path(file.stem).with_suffix(output_filetype)  # type: ignore[arg-type] # if output is defined, so is output_filetype
+            output / pathlib.Path(file.stem).with_suffix(output_filetype)
             if output
             else None
         )
@@ -302,15 +292,6 @@ def _run_file(
     if output is not None:
         writers.OrchestratorResults.validate_output(output=output)
 
-    parameters_dictionary = {
-        "thresholds": list(thresholds),
-        "calibrator": calibrator,
-        "epoch_length": epoch_length,
-        "activity_metric": activity_metric,
-        "nonwear_algorithm": list(nonwear_algorithm),
-        "input_file": str(input),
-    }
-
     if calibrator is not None and calibrator not in ["ggir", "gradient"]:
         msg = (
             f"Invalid calibrator: {calibrator}. Choose: 'ggir', 'gradient'. "
@@ -352,17 +333,14 @@ def _run_file(
         dynamic_range=watch_data.dynamic_range,
     )
 
-    sleep_detector = analytics.GgirSleepDetection(anglez)
-    sleep_windows = sleep_detector.run_sleep_detection()
-
-    nonwear_array = nonwear_utils.get_nonwear_measurements(
+    nonwear_array = processing_utils.get_nonwear_measurements(
         calibrated_acceleration=calibrated_acceleration,
         temperature=watch_data.temperature,
         non_wear_algorithms=nonwear_algorithm,
     )
 
-    nonwear_epoch = nonwear_utils.nonwear_array_cleanup(
-        nonwear_array=nonwear_array,
+    nonwear_epoch = processing_utils.synchronize_measurements(
+        data_measurement=nonwear_array,
         reference_measurement=activity_measurement,
         epoch_length=epoch_length,
     )
@@ -371,14 +349,40 @@ def _run_file(
         activity_measurement, thresholds
     )
 
+    sleep_detector = analytics.GgirSleepDetection(anglez)
+    sleep_parameters = sleep_detector.run_sleep_detection()
     sleep_array = analytics.sleep_cleanup(
-        sleep_windows=sleep_windows, nonwear_measurement=nonwear_epoch
+        sleep_windows=sleep_parameters.sleep_windows, nonwear_measurement=nonwear_epoch
+    )
+    spt_windows = analytics.sleep_bouts_cleanup(
+        sleep_parameter=sleep_parameters.spt_windows,
+        nonwear_measurement=nonwear_epoch,
+        time_reference_measurement=activity_measurement,
+        epoch_length=epoch_length,
     )
+    sib_periods = analytics.sleep_bouts_cleanup(
+        sleep_parameter=sleep_parameters.sib_periods,
+        nonwear_measurement=nonwear_epoch,
+        time_reference_measurement=activity_measurement,
+        epoch_length=epoch_length,
+    )
+
+    parameters_dictionary = {
+        "thresholds": list(thresholds),
+        "calibrator": calibrator,
+        "epoch_length": epoch_length,
+        "activity_metric": activity_metric,
+        "nonwear_algorithm": list(nonwear_algorithm),
+        "input_file": str(input),
+    }
+
     results = writers.OrchestratorResults(
         physical_activity_metric=activity_measurement,
         anglez=anglez,
         physical_activity_levels=physical_activity_levels,
         sleep_status=sleep_array,
+        sib_periods=sib_periods,
+        spt_periods=spt_windows,
         nonwear_status=nonwear_epoch,
         processing_params=parameters_dictionary,
     )

src/wristpy/io/writers/writers.py

@@ -23,6 +23,8 @@ class OrchestratorResults(pydantic.BaseModel):
     physical_activity_levels: models.Measurement
     nonwear_status: models.Measurement
     sleep_status: models.Measurement
+    sib_periods: models.Measurement
+    spt_periods: models.Measurement
     processing_params: Optional[Dict[str, Any]] = None
 
     def save_results(self, output: pathlib.Path) -> None: