CNT-codehub-pennsieve_data_stream

.gitmodules

@@ -34,3 +34,6 @@
 [submodule "core_libraries/submodules/networkx"]
 	path = core_libraries/submodules/networkx
 	url = https://github.com/networkx/networkx.git
+[submodule "scripts/codehub/utils/acquisition/BIDS"]
+	path = scripts/codehub/utils/acquisition/BIDS
+	url = https://github.com/penn-cnt/EEG_BIDS.git

README.md

@@ -6,14 +6,16 @@ CNT Code Hub
 
 This code is designed to help with the processing of epilepsy datasets commonly used within the Center for Neuroengineering & Therapeutics (CNT) at the University of Pennsylvania. 
 
-This code is meant to be researcher driven, allowing new code libraries to be added to modules that represent common research tasks (i.e. Channel Cleaning, Montaging, Preprocessing, etc.). The code can be accessed both as independent libraries that can be called on for a range of tasks, or as part of a large framework meant to ingest, clean, and prepare data for analysis or deep-learning tasks.
+This code is meant to be researcher driven, allowing new code libraries to be added to modules that represent common research tasks (i.e. Channel Cleaning, Montaging, Preprocessing, Feature Extraction, etc.). The code can be accessed both as independent libraries that can be called on for a range of tasks, or as part of a large framework meant to ingest, clean, and prepare data for analysis or deep-learning tasks.
 
-For more information on how to use our code, please see the examples folder for specific use-cases and common practices.
+We also provide a number of additional scripts to help with common/important tasks. For more information, please refer [here](https://github.com/penn-cnt/CNT-codehub/tree/main/scripts/codehub/utils/) for what scripts are currently available. 
 
-# Prerequisites
+# Installation
+
+## Prerequisites
 In order to use this repository, you must have access to Python 3+. You must also have access to conda 23.+ if building environments from yaml files.
 
-# Installation
+## Using Conda
 
 An environment file with all the needed packages to run this suite of code can be found at the following location:
 
@@ -39,17 +41,38 @@ The environment is then activated by running:
 
 More information about creating conda environments can be found [here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).
 
-## Adding the codehub to your python paths
+### Adding the codehub to your conda environment paths
 
 You will likely need to add this package to your python path to ensure full functionality of utility scripts and the main pipeline. To do so using anaconda, you can run:
 
 > conda develop <path-to-git-head>/scripts/codehub/
 
-For a virtual environment, an easy way to add `<path-to-git-head>/scripts/codehub/` to your path would be to add a text file with a .pth extention (any filename is fine) to the site-packages subfolder in your virtual environment folder. Within the text file you can just copy and paste the absolute path as the only contents.
+## Using venv and pip
+
+To create a virtual environment, you need to create a location for the environment to install to. For this example, we will specify `/demonstration/environment/cnt_codehub` as our environment location. Using the python version of your choice, in this example we will select 3.10, run the following command:
+
+> python3.10 -m venv /demonstration/environment/cnt_codehub
+
+to create a new virtual environment. To enter the envrionment, simply run:
+
+> source /demonstration/environment/cnt_codehub/bin/activate
 
-## Installation using venv
+**NOTE:** To streamline the process, we recommend making an alias command to avoid having to navigate the the activate file everytime.
+
+Once in the environment, a requirements.txt file with all the needed packages to run this suite of code can be found at the following location:
+
+> [CNT Codehub YAML](core_libraries/python/cnt_codehub/envs/requirements.txt)
+
+This file can be installed using the following call to pip from the envs subdirectory:
+
+> pip install -r requirements.txt
+
+which will install everything to your current virual environment. 
+
+### Adding the codehub to your virtual environment path
+For a virtual environment, an easy way to add `<path-to-git-head>/scripts/codehub/` to your path would be to add a text file with a .pth extention (any filename is fine) to the site-packages subfolder in your virtual environment folder. Within the text file you can just copy and paste the absolute path as the only contents.
 
-To be added soon.
+Typically, the path your your site-packages can be found at: `<path-to-environment-folder>/lib/python<version-number>/site-packages`.
 
 # Documentation
 

core_libraries/python/cnt_codehub/cnt_codehub.yml

@@ -0,0 +1,27 @@
+name: codehub
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - fooof
+  - neurodsp
+  - nltk
+  - numpy
+  - pandas
+  - prettytable
+  - pybids
+  - pytorch
+  - pyyaml
+  - scikit-learn
+  - scipy
+  - tqdm
+  - pip:
+      - ./wheels/ieeg-1.6-py3-none-any.whl
+      - edflib-python
+      - edfio
+      - mne
+      - mne-bids
+      - mne-icalabel
+      - nibabel
+      - pyEDFlib
+      - yasa

core_libraries/python/cnt_codehub/requirements.txt

@@ -0,0 +1,117 @@
+antropy==0.1.9
+astor
+asttokens==3.0.0
+attrs
+bids-validator
+bidsschematools
+cached-property
+certifi==2025.1.31
+charset-normalizer==3.4.1
+click
+colorama
+comm==0.2.2
+contourpy
+cycler
+decorator==5.2.1
+deprecation==2.1.0
+docopt
+edfio==0.4.8
+EDFlib-Python==1.0.8
+executing==2.2.0
+filelock
+fonttools
+fooof
+formulaic
+frozendict
+fsspec
+gmpy2
+graphlib-backport
+greenlet
+idna==3.10
+./wheels/ieeg-1.6-py3-none-any.whl
+importlib_resources
+interface_meta
+ipython
+ipython_pygments_lexers==1.1.1
+ipywidgets==8.1.6
+jedi==0.19.2
+Jinja2
+joblib
+jsonschema
+jsonschema-specifications
+jupyterlab_widgets==3.0.14
+kiwisolver
+lazy_loader==0.4
+lightgbm==4.6.0
+llvmlite
+lspopt==1.4.0
+markdown-it-py
+MarkupSafe
+matplotlib
+matplotlib-inline
+mdurl
+mne
+mne-bids
+mne-icalabel
+mpmath
+munkres==1.1.4
+networkx
+neurodsp
+nibabel
+nltk
+num2words
+numba
+numpy
+optree
+packaging
+pandas
+parso==0.8.4
+pennprov==2.2.4
+pexpect==4.9.0
+pillow
+pkgutil_resolve_name
+platformdirs==4.3.7
+pooch==1.8.2
+prettytable
+prompt_toolkit==3.0.51
+psutil==7.0.0
+ptyprocess==0.7.0
+pure_eval==0.2.3
+pybids
+pybind11
+pybind11_global
+pyEDFlib==0.1.40
+Pygments==2.19.1
+pyparsing
+pyriemann==0.8
+python-dateutil
+pytz
+PyYAML
+referencing
+regex
+requests==2.32.3
+rpds-py
+scikit-learn
+scipy
+seaborn==0.13.2
+setuptools==75.8.2
+six
+sleepecg
+SQLAlchemy
+stack-data==0.6.3
+sympy
+tabulate
+tensorpac==0.6.5
+threadpoolctl
+torch
+tqdm
+traitlets==5.14.3
+typing_extensions
+tzdata
+universal_pathlib
+urllib3==2.4.0
+wcwidth
+widgetsnbextension==4.0.14
+wrapt
+yasa==0.6.5
+zipp

scripts/CNT-STUDENT-DIRECTORY-GOES-HERE/README.md

@@ -0,0 +1,31 @@
+# Sample User Directory
+
+This is a placeholder folder to help you orient yourself to the repository for the first time. You can either rename this to your eventual project folder, or just delete it.
+
+We provide some information on how to make a project folder/rename this folder below.
+
+## Naming conventions
+
+This is an example user folder. Personal project work takes place in folders saved at the same level as the central `codehub' folder.
+
+We require that the naming of the folder follow the following design pattern:
+
+> {device name}\_{project name}\_{optional subdirectory}
+
+We require this naming structure in order to join different projects within the CNT data ecosystem. Multiple users can work within a single repository, either within the same directory or within their own optional subdirectories.
+
+### Example
+
+If I am working on a scalp multi-layer perceptron (MLP) project to predict sleep stages, spikes, and pnes predictions. I might go about making folders like follows:
+
+- `device_name`: I am working on scalp data, so I will go with `scalp`
+- `project_name`: I am using MLP for a a few different tasks, so lets just summarize the project as `MLP`.
+
+Now I could stop there and just make my folder: `scalp_MLP` and place all of my work within. Or, if I wanted to be careful about the environment for each sub-goal, or maybe I was collaborating and each person was doing their own sub-goal, I could make the following folders:
+- scalp_MLP_sleep
+- scalp_MLP_sleep-stags
+- scalp_MLP_pnes-predictor
+
+## Updating the codehub libraries
+
+Any changes to scripts within the [modules](../codehub/modules) subdiretory can be submitted to the main lab repository as its own branch, at which point a pull request will be reviewed before changes are accepted or rejected.

scripts/codehub/README.md

@@ -45,4 +45,9 @@ Public components are where lab code can be saved for everyone to use and for ep
 The utility scripts are not built into the epipy framework, and do not require specific formatting. To add a utility script, simply identify or create a new folder that generally defines the task being done (data acquisition/data validation/etc.) and add your code to the existing folder for that task type, or create a new folder defining the task and add it there.
 
 ### Pull Requests
-Submit a pull request to share your changes with the lab as a whole. The data team will review the request before merging it, or sending it back to you for more clarity or bug fixes.
+Submit a pull request to share your changes with the lab as a whole. The data team will review the request before merging it, or sending it back to you for more clarity or bug fixes.
+
+
+## Remaining Updates/Fixes
+
+- Remove keys from metadata handler if they do not get promoted to the feature extraction step.

scripts/codehub/allowed_arguments.yaml

@@ -16,6 +16,7 @@ allowed_channel_args:
     NEUROVISTA: Channels for NeuroVista data.
     RAW: Use all possible channels. Warning, channels may not match across different datasets.
 allowed_montage_args:
+    NONE: No montage.
     HUP1020: Use a 10-20 montage.
     NEUROVISTA: Use a custom NeuroVista montage.
     COMMON_AVERAGE: Use a common average montage.

scripts/codehub/components/core/internal/config_loader.py

@@ -27,7 +27,9 @@ def __init__(self,input_file):
             self.yaml_step = config
         else:
             # Read in and typecast the yaml file
-            config = yaml.safe_load(open(input_file,'r'))
+            fp     = open(input_file,'r')
+            config = yaml.safe_load(fp)
+            fp.close()
 
             # Add in any looped steps to the correct yaml input format
             self.loop_handler(config)

scripts/codehub/components/core/internal/dataframe_manager.py

@@ -39,16 +39,4 @@ def column_subsection(self,keep_columns):
 
         # Get the columns to drop
         drop_cols  = np.setdiff1d(self.dataframe.columns,keep_columns)
-        self.dataframe = self.dataframe.drop(drop_cols, axis=1)
-
-    def montaged_dataframe(self,data,columns):
-        """
-        Create a dataframe that stores the montaged data.
-        DEPRECIATE AFTER BETA PIPELINE RELEASE!
-
-        Args:
-            data (array): array of montaged data
-            columns (list): List of column names
-        """
-
-        self.montaged_dataframe = PD.DataFrame(data,columns=columns)
+        self.dataframe = self.dataframe.drop(drop_cols, axis=1)