Folder organisations

[lionelkusch]

From the PR #73, we need to have a vision about the organization of the library.

[lionelkusch]

My detail vision of it is:

.
+--ReadMe.txt
+--pyproject.toml
+--LICENCE
+--codecov.yml
+--.gitignore
+--doc_conf 
|  +--conf.py (configuration file)
|  +--Makefile (make file for creating the library)
|  +--documentation (folder for additional documentation)
|       +--api.rst (index of the API)
|       +--index.rst (home page of the documentation)
|       +--... (specific pages such as how to contribute, ....)
|       +--references.bib (bibliography reference)
|  +--docs (folder where to build the documentation)
+--examples (folder with examples)
|  +--__init__.py
|  +--figures (folder to save all the generated figures and additional figures)
|      +--generates (folders which contains generated by examples)
|         +--get_started
|              +--find_importance_variable_1.png
|              +--...
|         +--models
|              +--ada_svr_1.png (illustrate the data of the toy_dataset)
|              +--ada_svr_2.png (illustrate the results on the toy_dataset)
|              +--...
|         +--estimators 
|              +--dnn_learner_1.png 
|              +--...
|         +--comparison_models
|              +--geometric_2D_1.png
|              +--benchmarks_1D_dataset_1.png
|              +--...
|      +--external (folder which contains figures to illustrate the examples)
|         +--get_started
|              +--...
|         +--models
|              +--ada_svr_1.py (figure 1of Gaonkar et al. 2012)
|              +--...
|         +--comparison_models
|              +--...
|  +--_utils (folder with the function for examples)
|      +--__init__.py
|      +--plot_function.py (functions for plotting datasets or results)
|      +--....
|  +--get_started (folder for having tutorials or basic examples of how to use Hidimstat)
|      +--find_importance_variable.py (apply one generic method to a dataset)
|      +--....
|  +--models (folder which shows how to use models on toy dataset
                        and after a description of the methods
                        and a description of advantages, disadvantages and assumptions)
|      +--ada_svr.py 
|      +--permutation_test.py
|      +--.....
|  +--estimator (folder which shows how to use an estimator on toy dataset
                        and after a description of the estimator
                        and a description of advantages, disadvantages and assumptions)
|      +--dnn_learner.py 
|      +--.....
|  +--comparison_models (folder where the models are compared 
                                               on toy dataset with a specific characteristic
                                               (linear/non-linear, geometric, ...)
                                               and benchmark)
|      +--geometric_2D.py
|      +--....
|      +--benchmarks (study the speed of each model)
|          +--1D_dataset.py
|          +--scalability.py
|          +--performance p~n.py
|          +--....
+--hidimstat (folder with code)
|  +--__init__.py
|  +--models (folder with models for estimate variables of importance)
|      +--__init__.py
|      +--_utils (folder with functions shared between functions of the sub-packages)
|          +--__init__.py
|          +--scikit-learn_estimator.py (generic function for using the estimator API)
|          +--....
|      +--tests (folder for testing the models)
|          +--__init__.py
|          +--_utils (folder with functions shared between tests)
|              +--__init__.py
|              +--....
|          +--test_ada_svr.py
|          +--test_permutation_test.py
|          +--....
|      +--ada_svr.py
|      +--permutation_test.py
|      +--....
|  +--estimator (folder with specific estimators)
|      +--__init__.py
|      +--_utils (folder with functions shared between functions of the sub-packages)
|          +--__init__.py
|          +--....
|      +--tests (folder for testing the estimators)
|          +--__init__.py
|          +--_utils (folder with functions shared between tests)
|              +--__init__.py
|              +--....
|          +--test_dnn_learner.py
|          +--....
|      +--dnn_learner.py
|      +--....
|  +--extra (folder for generation of toy_dataset and statistics methods)
|      +--__init__.py
|      +--toy_data
|          +--__init__.py
|          +--_utils (folder with functions shared between functions of the subsub-packages)
|              +--__init__.py
|              +--....
|          +--tests (folder for testing the generated function)
|              +--__init__.py
|              +--_utils (folder with functions shared between tests)
|                  +--__init__.py
|                  +--....
|              +--test_1d_dataset.py
|              +--....
|          +--1d_dataset.py
|          +--....
|      +--stat_tools (the folder contains methods for calculating pvalue and calculating error rate)
|          +--__init__.py
|          +--_utils (folder with functions shared between functions of the subsub-packages)
|              +--__init__.py
|              +--....
|          +--tests (folder for testing the generated function)
|              +--__init__.py
|              +--_utils (folder with functions shared between tests)
|                  +--__init__.py
|                  +--....
|              +--test_pval.py
|              +--....
|          +--pval.py
|          +--....

[lionelkusch]

It will be impossible to talk about this very detailed vision.
To summarise:

• doc_conf (contains all the documentation)
• examples (contains all the examples)
• hidimstat/models (contains all the models )
• hidimstat/estimators (contains all the estimators)
• hidimstat/extra (contains functions for the generation of the dataset and statistics methods (pvalue calculus, calculus of error of type I, type II, ...)

[lionelkusch]

The doc_conf is composed of:

• configuration files and makefile
• additional documentation pages

For the moment, there is not a specific organisation of the different nonautogenerated documentation.

[lionelkusch]

The examples folders is composed of 5 folders:

• figures: organize the generated figure for the documentation and store the figure for helping to understand the examples
• get_started: very basic usage of the library
• models: example of usage models with some explanation of it
• estimators: example of usage estimators with some explanation of it
• comparison_models: example of comparison models and benchmarks

[lionelkusch]

hidimstat is composed of 3 sub-packages:

• estimators
• models
• extra/toy_data (function for the generation of toy_dataset)
• extra/stat_tools (functions for statistical tools)

If sub_packages are composed of a _utils folder (shared function in subpackage), a test folder (for the tests) and the functions.

[lionelkusch]

@bthirion @Remi-Gau @jpaillard @man-shu
If you have any suggestions on how to organize the files, I had love to hear from you.

[Remi-Gau]

quick things:

• adding init.py and utils in examples: won't that make the examples hard to run for users?
• since the library is not huge (yet) consider moving to a source layout:
  • https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/
  • that usually goes along with putting the tests outside of the source code: https://docs.pytest.org/en/stable/explanation/goodpractices.html#tests-outside-application-code

[Remi-Gau]

See also: https://learn.scientific-python.org/development/guides/pytest/#tests-should-test-the-installed-version-not-the-local-version

[jpaillard]

I have a few suggestions:

• The names 'models' and 'estimator' can be confusing since they are sometimes used interchangeably. A more explicit name like 'stat_models' and 'prediction_models' could be more transparent.
• For the toy datasets, having a datasets folder would be clearer; it's organized this way in Sklearn and Nilearn
• That could avoid having the 'extra' folder, which is not so straightforward. Instead, we have 'datasets' and 'stat_tools.'

[lionelkusch]

The toy_dataset won't contain data; it will contain only functions for generating data, in my view.
I don't think it's a good idea to have a dataset for the moment.

[man-shu]

My suggestions:

• I would keep the names of submodules short (without _). I agree that names models and estimators are confusing. I could suggest alternative names for models and estimators, but I don't understand the difference between the two in the context of the package. Maybe you could point me to some code?
• I think it would be good to keep toy_datasets and stat_tools separate as they don't seem to be related and also rename to datasets and stats. Correct me if I am wrong, but I don't think you would provide actual datasets with the package, they would still be the code to fetch some real datasets, so datasets seems appropriate to me.

[Remi-Gau]

• I would keep the names of submodules short (without _)

except for the _utils folder that make sense when you want to keep things private to a subpackage

2 tools I have used in other projects:

• https://pypi.org/project/flake8-private-name-import helps make sure that you don't import private things where they should not be imported
• https://import-linter.readthedocs.io/en/stable/readme.html#overview allows to make sure that you keep strict architecture in your package by establishing rules on what each module can import from, can help establish what modules are more 'low level' than others