ehrapy transition to ehrdata: take over ehrapys data functionalities (#136)

* add basic dataframe support and test

* more tests, modified to_dataframe

* read_csv

* save commit

* more on csv, h5ad

* save commit

* updates, expanded from_pandas

* expand to_pandas

* type inference and missing value as strings

* iter commit, tests fail

* fix to_pandas and feature types, add tests

* add tests and doc for four new datasets

* save commit

* basic draft for read_zarr

* h5ad, zarr io draft

* harmonize to filename as anndata

* remove commented out code

* changelog, refactoring

* unreleased statement in changelog

* fix refactoring

* include feature type functions in doc:

* be more clear about file formats in download internally

* Apply suggestions from code review

Co-authored-by: Lukas Heumos <lukas.heumos@posteo.net>

* more intersphinx, more pandas examples, fixes

* iter commit

* address more review comments:

* sparse io test files

* cleaning round, reduce redundancy

* add missing h5ad, argument cleaning

* pandas description in tutorial

* remove backed comment zarr

* more explicit arguments, cleanup

* test nonnumeric layer

* utilzz refactor

* fix doc? fix typos

* refactor .tl, .io, fix typos

* fix display of first level functions?

* Apply suggestions from code review

Co-authored-by: Lukas Heumos <lukas.heumos@posteo.net>

* address comments

* split lines in csv

* split lins for more docstrings in io

* hdf5, fastarrayutils dependency

* csv read description like pandas

* more io like pandas wording

---------

Co-authored-by: Lukas Heumos <lukas.heumos@posteo.net>

Author: eroell

.gitignore

@@ -16,6 +16,7 @@ __pycache__/
 /node_modules/
 
 # docs
+/docs/tutorials/data/
 /docs/generated/
 /docs/_build/
 /docs/generated/

CHANGELOG.md

@@ -8,11 +8,36 @@ and this project adheres to [Semantic Versioning][].
 [keep a changelog]: https://keepachangelog.com/en/1.0.0/
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
+## [0.0.6] Not yet released
+
+### Fixed
+- Cleaned up and updated tutorial notebooks ([#140](https://github.com/theislab/ehrdata/pull/140)) @agerardy
+
+### Added
+- {func}`~ehrdata.io.read_csv` Reads a csv file ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.read_h5ad` Reads an h5ad file ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.read_zarr` Reads a zarr file ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.write_h5ad` Writes an h5ad file ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.write_zarr` Writes a zarr file ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.from_pandas` Transform a given {class}`~pandas.DataFrame` into an {class}`~ehrdata.EHRData` object ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.io.to_pandas` Transform an {class}`~ehrdata.EHRData` object into a {class}`~pandas.DataFrame` ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.dt.mimic_2` Loads the MIMIC-II dataset ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.dt.mimic_2_preprocessed` Loads the preprocessed MIMIC-II dataset ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.dt.diabetes_130_raw` Loads the raw diabetes-130 dataset ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.dt.diabetes_130_fairlearn` Loads the preprocessed diabetes-130 dataset by fairlearn ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.infer_feature_types` Infer feature types in an {class}`~ehrdata.EHRData` object ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.feature_type_overview` Overview of inferred feature types ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.replace_feature_types` Replacing inferred feature types ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+- {func}`~ehrdata.harmonize_missing_values` Harmonize missing values in an {class}`~ehrdata.EHRData` object ([#136](https://github.com/theislab/ehrdata/pull/136)) @eroell
+
+
+### Modified
+
 ## [0.0.5]
 
 ### Fixed
 
-- Initialize EHRData with X and layers
+- Initialize EHRData with X and layers ([#132](https://github.com/theislab/ehrdata/pull/132)) @eroell
 
 ### Added
 

docs/api/datasets_index.md

@@ -15,4 +15,8 @@
     dt.gibleed_omop
     dt.synthea27nj_omop
     dt.physionet2012
+    dt.mimic_2
+    dt.mimic_2_preprocessed
+    dt.diabetes_130_raw
+    dt.diabetes_130_fairlearn
 ```

docs/api/ehrdata_index.md

@@ -12,4 +12,9 @@
 
     EHRData
 
+    infer_feature_types
+    feature_type_overview
+    replace_feature_types
+    harmonize_missing_values
+
 ```

docs/api/io_index.md

@@ -10,6 +10,13 @@
     :toctree: io
     :nosignatures:
 
+    io.read_csv
+    io.read_h5ad
+    io.read_zarr
+    io.write_h5ad
+    io.write_zarr
+    io.from_pandas
+    io.to_pandas
     io.omop.setup_connection
     io.omop.setup_obs
     io.omop.setup_variables

docs/conf.py

@@ -148,5 +148,6 @@
 # Redirect broken parameter annotation classes
 qualname_overrides = {
     "zarr._storage.store.Store": "zarr.storage.MemoryStore",
+    "zarr.core.group.Group": "zarr.group.Group",
     "lnschema_core.models.Artifact": "lamindb.Artifact",
 }

docs/references.bib

@@ -1,3 +1,19 @@
+@article{strack2014impact,
+  title={Impact of HbA1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records},
+  author={Strack, Beata and DeShazo, Jonathan P and Gennings, Chris and Olmo, Juan L and Ventura, Sebastian and Cios, Krzysztof J and Clore, John N},
+  journal={BioMed research international},
+  volume={2014},
+  number={1},
+  pages={781670},
+  year={2014},
+  publisher={Wiley Online Library}
+}
+@article{bird2020fairlearn,
+  title={Fairlearn: A toolkit for assessing and improving fairness in AI},
+  author={Bird, Sarah and Dud{\'\i}k, Miro and Edgar, Richard and Horn, Brandon and Lutz, Roman and Milan, Vanessa and Sameki, Mehrnoosh and Wallach, Hanna and Walker, Kathleen},
+  journal={Microsoft, Tech. Rep. MSR-TR-2020-32},
+  year={2020}
+}
 @article{Virshup_2023,
 	doi = {10.1038/s41587-023-01733-8},
 	url = {https://doi.org/10.1038%2Fs41587-023-01733-8},
@@ -8,3 +24,31 @@ @article{Virshup_2023
 	title = {The scverse project provides a computational ecosystem for single-cell omics data analysis},
 	journal = {Nature Biotechnology}
 }
+@book{critical2016secondary,
+  title={Secondary analysis of electronic health records},
+  author={Critical Data, MIT},
+  year={2016},
+  publisher={Springer Nature}
+}
+@article{du2023saits,
+  title={Saits: Self-attention-based imputation for time series},
+  author={Du, Wenjie and C{\^o}t{\'e}, David and Liu, Yan},
+  journal={Expert Systems with Applications},
+  volume={219},
+  pages={119619},
+  year={2023},
+  publisher={Elsevier}
+}
+@article{du2023pypots,
+  title={PyPOTS: a Python toolbox for data mining on Partially-Observed Time Series},
+  author={Du, Wenjie},
+  journal={arXiv preprint arXiv:2305.18811},
+  year={2023}
+}
+@article{kallfelz2021mimic,
+  title={MIMIC-IV demo data in the OMOP Common Data Model},
+  author={Kallfelz, Michael and Tsvetkova, Anna and Pollard, Tom and Kwong, Manlik and Lipori, Gigi and Huser, Vojtech and Osborn, Jeffrey and Hao, Sicheng and Williams, Andrew},
+  journal={object Object]. doi},
+  volume={10},
+  year={2021}
+}