Project Structure

[tachyonicClock]

Our project has gotten bigger and now supports different experimental settings, but the directory layout is inconsistently structured in places:

capymoa.datasets
capymoa.datasets.ocl
capymoa.evaluation.ocl
capymoa.ocl
capymoa.classifier
capymoa.regressor
capymoa.ssl.classifier
capymoa.drift.detectors
capymoa.drift.eval_detector

I propose a consistent layout capymoa.{ocl,ssl,drift,anomaly,classifier,regressor}.{__init__,datasets,base,evaluation}

• capymoa.*.__init__.py would contain learner classes. For example:

from capymoa.classifier import AdaptiveRandomForestClassifier

• capymoa.*.evaluation would contain evaluation functions and classes. For example:

from capymoa.classifier.evaluation import prequential_evaluation

• capymoa.*.datasets would contain classification datasets. For example:

from capymoa.classifier.datasets import ElectricityTiny

• capymoa.*.base would contain base classes of the experimental setting. For example:

from capymoa.classifier.base import Classifier

from capymoa.classifier import AdaptiveRandomForestClassifier
from capymoa.classifier.datasets import ElectricityTiny
from capymoa.classifier.evaluation import prequential_evaluation

stream = ElectricityTiny()
learner = AdaptiveRandomForestClassifier()
prequential_evaluation(stream, learner)

Common classes and functions would be moved to capymoa.core.{instance,stream} module.

Any thoughts?

[tachyonicClock]

At the moment ocl only supports classification but in the future we might want regression. We could add capymoa.ocl.{classifier,regressor}. But simply using capymoa.{ocl,ocl_regressor} might be better.

[hmgomes]

Very important questions to be discussed. One principle that we can follow is to make things as simple as possible, so less nesting whenever possible. Looking at sklearn API here, they have only one level per package, the rest they use naming conventions.

sklearn	capymoa	Level 2	Level 3
sklearn	capymoa.anomaly
sklearn.base	capymoa.automl
sklearn.calibration	capymoa.classifier
sklearn.cluster	capymoa.clusterers
sklearn.compose	capymoa.datasets
sklearn.covariance	capymoa.drift
sklearn.cross_decomposition		capymoa.drift.detectors
sklearn.datasets			capymoa.drift.detectors.abcd_components
sklearn.decomposition	capymoa.evaluation
sklearn.discriminant_analysis
sklearn.dummy	capymoa.ocl
sklearn.ensemble		capymoa.ocl.util
sklearn.exceptions	capymoa.prediction_interval
sklearn.experimental	capymoa.regressor
sklearn.feature_extraction	capymoa.ssl
sklearn.feature_selection		capymoa.ssl.classifier
sklearn.frozen	capymoa.stream
sklearn.gaussian_process		capymoa.stream.preprocessing
sklearn.impute
sklearn.inspection
sklearn.isotonic
sklearn.kernel_approximation
sklearn.kernel_ridge
sklearn.linear_model
sklearn.manifold
sklearn.metrics
sklearn.mixture
sklearn.model_selection
sklearn.multiclass
sklearn.multioutput
sklearn.naive_bayes
sklearn.neighbors
sklearn.neural_network
sklearn.pipeline
sklearn.preprocessing
sklearn.random_projection
sklearn.semi_supervised
sklearn.svm
sklearn.tree
sklearn.utils

[hmgomes]

One comment about this one:

I propose a consistent layout capymoa.{ocl,ssl,drift,anomaly,classifier,regressor}.{init,datasets,base,evaluation}

This means that each of them will contain an evaluation and datasets subpackage? E.g.

from capymoa.classifier.evaluation import prequential_evaluation

I understand this will make things predictable and organised, but some tasks share functions. For example, the high-level evaluation function prequential_evaluation was designed to work with classification/regression problems, thus maybe for things like datasets, stream generators and evaluation we can use the class name to distinguish? E.g. prequential_evaluation_anomaly instead of capymoa.anomaly.evaluation.prequential_evaluation

[hmgomes]

The OCL is an interesting problem to tackle. It is a tough decision whether we want to bring things (stream and OCL) close together or try to keep them apart. There are arguments in favour of both

At the moment ocl only supports classification but in the future we might want regression. We could add capymoa.ocl.{classifier,regressor}. But simply using capymoa.{ocl,ocl_regressor} might be better.

For example, someone doing OCL might want to use streaming algorithms for benchmarking (e.g. try out HTs on a simple problem). Some researchers have been referring to streaming problems (using structured data streams) as online continual learning. Maybe that is because in their problem setting forgetting is an issue, while normally in stream learning that is not the primary concern.