Update v1.7.0 (#21)

The 1.7.0 update introduces the new "rich" console mode, based on the Rich library, and refines the metrics output interface, allowing dynamic switching between modes without exess tech debt. The analysis target variable pattern has been completely overhauled to use a configurable object that handles all labels relevant to downstream task analyses, along with confounders, with finer-grained control. In addition, we introduce a timeout feature that allows long metrics to be interrupted and skipped, we improve error handling and configurability, and we add Maximum Mean Discrepancy, along with Feature Importance Overlap, to the metric library.

**Refactor:**
* Changed the `format_output` method in all metric classes to return a list of tuples suitable for rich console output, replacing previous string-based formatting. This affects the metric template, core metric class, and all metric implementations.
* Changed the `analysis_target_var` to an `analysis_target` object: parsing is handled automatically to retain the simple interface, but the object can also be parsed as an object for superior fine-grained control. See the new section in `guides/syntheval_guide.ipynb` for a tutorial on how this object can be used. 

**New Features:**
* Added a `console` argument for the main class, which allows specifying use of `rich` (new) or `ascii` (legacy) formatting for the console print or to turn `off` entirely. We added a check to automatically switch to `ascii` from `rich`if in a notebook environment to prevent crashing the terminal. 
* Added a timeout feature based on `asyncio`, for the main evaluation loop to allow interrupting and skipping of metrics that take too long to complete. By default, timeout is *not* enabled.
* Added a `plot_figures` attribute to metric classes, allowing users to control figure plotting separately from verbosity.
* Added a corresponding argument `enable_plots` to the SynthEval class, to control plotting in the main evaluation loop. 
* Added a `missing_directive` argument to the SynthEval class, so that the user can control if SynthEval should raise a warning, drop rows with missingness, or ignore that there is missingness and carry on. We added a small discussion in the `guides/preprocessing.md` guide on missingness, for users interested in other solutions. 
* Added the Maximum Mean Discrepancy (MMD) metric, recording both the biased and unbiased versions of the statistic. A detailed explanation and reference are added in `guides/metrics_references.md`. 
* Added the Feature Importance Overlap (FIO) metric, which checks two properties related to feature selection/ranking tasks. Namely, that the importance values assigned by a predictive model match (mean absolute error and weighted mean absolute error), and that the features recovered in a ranking at 5%, 10%, 25% and 50% of features are the same. The metric can also plot the top feature importance scores. A description of this metric is added in `guides/metrics_references.md`. 
* With the new `analysis_target` object, the increased flexibility allowed some important changes in the classification accuracy, auroc difference, attribute disclosure risk, and statistical parity metrics, now accounting for multiple potential labels, and dynamically removing confounders in prediction tasks (this is also considered in the new FIO metric).

**Documentation:** 
* Minor details were added to the `README.md`.
* Major reformatting of the metrics overview in `README.md` into tables with metric keywords and links to the method documentation, and in the few instances where applicable to the `guides/metrics_references.md`. 
* Guide codebooks were refreshed with the latest features, new metrics, and nicer printing. `guides/syntheval_guide.ipynb` now includes a new part on the `analysis_target` object.
* New `guides/preprocessing.md` added to document preprocessing steps. 
* Adjusted a number of the metric docstrings.

**Changes:**
* Improved error handling in several metric scripts by raising `ValueError` instead of printing warnings or passing on failed assertions, ensuring clearer feedback for users and that errors are raised in the active console.
* Changed default ranking system for the benchmark method, from `linear` (min-max sum) to `summation` (flat sum). 
* Changed the default preprocessing for PCA metric from "mean" to "std" for consistency. 
* Changed the default F1 setting in the classification metric from `micro`to `weighted`, for more saturation-aware behaviour on imbalanced classification problems. 
* Changed the default confidence interval unit for CIO and DWM metrics from `sem` to `std`. The new option `ci` can be used to switch back to the old behaviour if needed.
* Attribute disclosure metric had the `sensitive` argument removed; now the sensitive attributes are parsed through the `analysis_target`object.
* Statistical parity metric similarly had the `protected_attribute` argument removed, and now also uses the `sensitive_vars` attribute in the `analysis_target`object. In addition, the statistical parity metric can now evaluate multiple protected attributes.

**Bug fixes:**
* Hellinger distance metric had a division by zero error when determining binwidth when the interquartile range was 0: this error is now caught, and handles the binning in the non-monotonous case using variable binwidth. 
* Adding the timeout feature, caused a bunch of warnings from plotting outside of the main thread, on certain versions of matplotlib with tkinter; this is fixed by using the 'Agg' backend for plotting. In addition, the `async.io` caused a notebook crash on some versions, so we added a catch that replaces direct event-loop calls with a loop-self-helper using coroutines if a loop is already running. 
* Improved type casting to meet NumPy 2.x scalar handling (tests broke because of previously lazy type-handling)
* Attribute disclosure risk, membership inference attack, and Kolmogorov-Smirnov test metrics would return undefined STD for STD calculation on lists of length 1 (which was not a huge problem, but would throw warnings); we made catches to avoid this.

Author: notna07

.github/workflows/doctests.yml

@@ -25,7 +25,11 @@ jobs:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
-            python-version: ">=3.9"
+              python-version: ">=3.9"
+        - name: Install OpenBLAS
+          run: |
+            sudo apt-get update
+            sudo apt-get install -y libopenblas-dev
         - name: Install dependencies
           run: |
             python -m pip install --upgrade pip

.github/workflows/release.yml

@@ -11,7 +11,7 @@ name: Upload Python Package
 
 on:
   release:
-    types: [published]
+    types: [released, prereleased]
 
 permissions:
   contents: read

README.md

@@ -40,9 +40,9 @@ from syntheval import SynthEval
 evaluator = SynthEval(df_real, holdout_dataframe = df_test, cat_cols = class_cat_col)
 evaluator.evaluate(df_fake, class_lab_col, presets_file = "full_eval", **kwargs)
 ```
-Where the user supplies <code>df_real, df_test, df_fake</code> as pandas dataframes, the <code>class_cat_col</code> is a complete list of column names (which can be omitted for categoricals to be automatically inferred). Some metrics require a target class, so <code>class_lab_col</code> is a string for designating one column with discrete values as a target for usability predictions and colouration. In the evaluate function, a presets file can be chosen ("full_eval", "fast_eval", or "privacy") or alternatively, a filepath can be supplied to a json file with select metrics keywords. Finally, instead of (or in addition to), keyword arguments can be added in the end with additional metrics and their options. 
+Where the user supplies <code>df_real, df_test, df_fake</code> as pandas dataframes, the <code>class_cat_col</code> is a complete list of column names (which can be omitted for categoricals to be automatically inferred). Some metrics require a target class, so <code>class_lab_col</code> is a string (or list or [AnalysisConfig](guides/syntheval_guide.ipynb#Analysis-Target-Configuration) object) for designating variables as a target for downstream task prediction and plotting colouration. In the evaluate function, a presets file can be chosen ("full_eval", "fast_eval", or "privacy") or alternatively, a filepath can be supplied to a json file with select metrics keywords. Finally, instead of (or in addition to), keyword arguments can be added in the end with additional metrics and their options. 
 
-New in version 1.4 is the benchmark module, that allows a directory of synthetic datasets to be specified for evaluation (or a dictionary of dataframes). All datasets in the folder are evaluated against the training (and test) data on the selected metrics. Three types of rank-derived scoring are available to choose between ("linear", "normal", or "quantile"), assisting in identifying datasets that perform well overall, and on utility and privacy dimensions.
+Version 1.4 introduced the benchmark module, that allows a directory of synthetic datasets to be specified for evaluation (or a dictionary of dataframes). All datasets in the folder are evaluated against the training (and test) data on the selected metrics. Three types of rank-derived scoring are available to choose between ("linear", "normal", or "quantile"), assisting in identifying datasets that perform well overall, and on utility and privacy dimensions.
 ```python
 evaluator.benchmark('local/path_to/target_dir/', class_lab_col, presets_file = "full_eval", rank_strategy='normal', **kwargs)
 ```
@@ -54,6 +54,9 @@ For more details on how to use the library, see the codebooks below;
 | [Tutorial 1](guides/syntheval_guide.ipynb) | Get started, basic examples |
 | [Tutorial 2](guides/syntheval_benchmark.ipynb) | Dataset benchmark, evaluating and ranking synthetic datasets in bulk |
 | [Tutorial 3](https://github.com/schneiderkamplab/syntheval-model-benchmark-example/blob/main/syntheval_model_benchmark.ipynb) | Model benchmark example, evaluating and ranking models |
+| --- | --- |
+| [Preprocessing Reference](guides/preprocessing.md) | Documentation on data preprocessing steps |
+| [Metrics Reference](guides/metrics_references.md) | Documentation of the newer metrics that are not covered in the SynthEval paper |
 
 ### Command line interface
 SynthEval can also be run from the commandline with the following syntax:
@@ -76,46 +79,47 @@ Options:
 ## Included metrics overview
 The SynthEval library comes equipped with a broad selection of metrics to evaluate various aspects of synthetic tabular data. One of the more interesting properties that makes SynthEval stand out is that many of the metrics have been carefully adapted to accept heterogeneous data. Distances between datapoints are (by default) handled using Gower's distance/similarity measure rather than the Euclidean distance, which negates any requirement of special data encoding.
 
+The metrics are divided into three categories, utility, privacy and fairness, and the results are reported in a standardised format with an average value and an error estimate (where applicable). In addition to using the default preset files, users can select specific metrics and options by using the metric keywords in the evaluate function. The following table gives an overview of the available metrics, their keywords, and links to documentation for each metric.
+
 ### Utility Metrics
 Utility analysis entails resemblance, quality and usability metrics testing how well the synthetic data looks like, behaves like, and substitutes like the real data.
 
-In the code we implemented:
-- Dimension-Wise Means (nums. only, avg. value and plot)
-- Principal Components Analysis (nums. only, plot of first two components)
-- Confidence Interval Overlap (nums. only, number and fraction of significant tests)
-- Correlation Matrix Difference (mixed correlation)
-- Mutual Information Matrix Difference
-- Kolmogorov–Smirnov / Total Variation Distance test (avg. distance, avg. p-value and number and fraction of significant tests)
-- Hellinger Distance (avg. distance)
-- Propensity Mean Squared Error (pMSE and accuracy)
-- Prediction AUROC difference (for binary target variables only)
-- Nearest Neighbour Adversarial Accuracy (NNAA) 
-
-### classification accuracy
-In this tool we test useability by training four different <code>sklearn</code> classifiers on real and synthetic data with 5-fold cross-validation (testing both models on the real validation fold). 
-- DecisionTreeClassifier
-- AdaBoostClassifier
-- RandomForestClassifier
-- LogisticRegression
-
-The average accuracy is reported together with the accuracy difference from models trained on real and synthetic data. If a test set is provided, the classifiers are also trained once on the entire training set, and again the accuracy and accuracy differences are reported, but now on the test data.
-
-By default the results are given in terms of accuracy (micro F1 scores). To change, use ‘micro’, ‘macro’ or ‘weighted’ in the preset file or in kwargs.
+| keyword | metric name | link to docs | description | 
+| --- | --- | --- | --- |
+| `dwm` | Dimension-Wise Means | [DimensionWiseMeans](src\syntheval\metrics\utility\metric_dimensionwise_means.py) | nums. only, avg. value and plot |
+| `pca` | Principal Components Analysis | [PrincipalComponentsAnalysis](src\syntheval\metrics\utility\metric_principal_component_analysis.py) | [Text Documentation](guides/metrics_references.md#Inter-Dataset-Similarity-Metric-Based-on-PCA) |
+| `cio` | Confidence Interval Overlap | [ConfidenceIntervalOverlap](src\syntheval\metrics\utility\metric_confidence_interval_overlap.py) | nums. only, number and fraction of significant tests |
+| `corr_diff` | Correlation Matrix Difference | [MixedCorrelation](src\syntheval\metrics\utility\metric_mixed_correlation.py) | mixed correlation |
+| `mi_diff` | Mutual Information Matrix Difference | [MutualInformation](src\syntheval\metrics\utility\metric_mutual_information.py) | mixed correlation |
+| `ks_test` | Kolmogorov–Smirnov / Total Variation Distance test | [KolmogorovSmirnov](src\syntheval\metrics\utility\metric_kolmogorov_smirnov.py) | avg. distance, avg. p-value and number and fraction of significant tests |
+| `h_dist` | Hellinger Distance | [HellingerDistance](src\syntheval\metrics\utility\metric_hellinger_distance.py) | avg. distance |
+| `p_MSE` | Propensity Mean Squared Error | [PropensityMeanSquaredError](src\syntheval\metrics\utility\metric_propensity_mse.py) | pMSE and accuracy |
+| `auroc_diff` | Prediction AUROC difference | [PredictionAUROCDifference](src\syntheval\metrics\utility\metric_auroc_difference.py) | for binary target variables only |
+| `cls_acc`| Classification Accuracy | [ClassificationAccuracy](src\syntheval\metrics\utility\metric_accuracy_difference.py) | avg. TRTR, TSTR across four classifiers, with optional holdout data and 5-fold cross-validation |
+| `fio` | Feature Importance Overlap | [FeatureImportanceOverlap](src\syntheval\metrics\utility\metric_feature_importance_overlap.py) | [Text Documentation](guides/metrics_references.md#Feature-Importance-Overlap-(FIO)) |
+| `nnaa` | Nearest Neighbour Adversarial Accuracy | [NearestNeighbourAdversarialAccuracy](src\syntheval\metrics\privacy\metric_nn_adversarial_accuracy.py) | avg. NNAA across all records |
+| `q_mse` | Quantile MSE | [QuantileMSE](src\syntheval\metrics\utility\metric_quantile_mse.py) | [Text Documentation](guides/metrics_references.md#Quantile-MSE) |
+| `mmd` | Maximum Mean Discrepancy (MMD) | [MaximumMeanDiscrepancy](src\syntheval\metrics\utility\metric_max_mean_discrepancy.py) | [Text Documentation](guides/metrics_references.md#Maximum-Mean-Discrepancy-(MMD)) |
+
 
 ### Privacy Metrics
 Privacy is a crucial aspect of evaluating synthetic data, we include only three highlevel metrics with more to be added in the future.
-- Nearest Neighbour Distance Ratio (NNDR)
-- Privacy Losses (difference in NNAA and NNDR between test and training sets, good for checking overfitting too.)
-- Median Distance to Closest Record (normalised by internal NN distance.)
-- Hitting Rate (for numericals defined to be within the attribute range / 30)
-- Epsilon identifiability risk (calculated using weighted NN distance)
-- Membership Inference Attack
-- Attribute Disclosure Risk (with or without holdout data)
+
+| keyword | metric name | link to docs | description | 
+| --- | --- | --- | --- |
+| `nndr` | Nearest Neighbour Distance Ratio | [NearestNeighbourDistanceRatio](src\syntheval\metrics\privacy\metric_nn_distance_ratio.py) | avg. NNDR across all records |
+| `dcr` | Median Distance to Closest Record | [MedianDistanceToClosestRecord](src\syntheval\metrics\privacy\metric_distance_closest_record.py) | normalised by internal NN distance |
+| `hit_rate` | Hitting Rate | [HittingRate](src\syntheval\metrics\privacy\metric_hitting_rate.py) | hits on numericals are within attribute range / 30 |
+| `eps_risk` | Epsilon Identifiability Risk | [EpsilonIdentifiabilityRisk](src\syntheval\metrics\privacy\metric_epsilon_identifiability.py) | calculated using weighted NN distance |
+| `mia` | Membership Inference Attack | [MIAClassifier](src\syntheval\metrics\privacy\metric_MIA_classification.py) | worst case adversarial knowledge attack |
+| `att_discl` | Attribute Disclosure Risk | [AttributeDisclosure](src\syntheval\metrics\privacy\metric_AttrDis.py) | with or without holdout data | 
 
 ### Fairness Metrics
 Fairness is an emerging property of synthetic data, we recently added support to evaluate this aspect, and include for now:
-- Statistical Parity Difference (Also known as Demographic Parity)
 
+| keyword | metric name | link to docs | description | 
+| --- | --- | --- | --- |
+| `statistical_parity` | Statistical Parity Difference | [StatisticalParity](src\syntheval\metrics\fairness\metric_statistical_parity.py) | also known as Demographic Parity |
 
 ## Creating new metrics
 SynthEval is designed with modularity in mind. Creating new, custom metrics is as easy as copying the [metrics template file](https://github.com/schneiderkamplab/syntheval/blob/main/src/syntheval/metrics/metric_template.py), and filling in the five required functions. Because SynthEval has very little hardcoding wrt. the metrics, making new metrics work locally should require no changes other than adding the metrics script in the metrics folder.

metrics_references.md guides/metrics_references.mdmetrics_references.md renamed to guides/metrics_references.md

@@ -26,4 +26,33 @@ $$
 where $x_j$ is the estimated probability in each of the $N_{quant}$ quantiles. The metric is calculated for each column in the data, and the average is taken over all columns. The metric is used to evaluate the distribution of the synthetic data, and a low value indicates that the synthetic data is a good representation of the real data. 
 
 Reference:
-> Butter, A., Diefenbacher, S., Kasieczka, G., Nachman, B., & Plehn, T. (2021). GANplifying event samples. SciPost Physics, 10(6), 139. [10.21468/SciPostPhys.10.6.139](https://doi.org/10.21468/SciPostPhys.10.6.139)
+> Butter, A., Diefenbacher, S., Kasieczka, G., Nachman, B., & Plehn, T. (2021). GANplifying event samples. SciPost Physics, 10(6), 139. [10.21468/SciPostPhys.10.6.139](https://doi.org/10.21468/SciPostPhys.10.6.139)
+
+### Maximum Mean Discrepancy (MMD)
+MMD is a kernel-based distance measure between two distributions (in our case the real and synthetic data sets). It comes in two flavors: the biased V-statistic and the unbiased U-statistic. The biased V-statistic is calculated as follows:
+
+$$
+\text{MMD}_b^2 = \frac{1}{n^2} \sum_{i,j} k(x_i, x_j) + \frac{1}{m^2} \sum_{i,j} k(y_i, y_j) - \frac{2}{nm} \sum_{i,j} k(x_i, y_j),
+$$
+
+where $k$ is a kernel function, and $x_i$ and $y_j$ are samples from the two distributions. The unbiased U-statistic is calculated as follows:
+
+$$
+\text{MMD}_u^2 = \frac{1}{n(n-1)} \sum_{i \neq j} k(x_i, x_j) + \frac{1}{m(m-1)} \sum_{i \neq j} k(y_i, y_j) - \frac{2}{nm} \sum_{i,j} k(x_i, y_j),
+$$
+
+where the sums are taken over all pairs of samples, excluding the diagonal terms. Both measures can be negative at finite sample sizes due to variance, so it is clipped at $0$ for stability, i.e., $\max(MMD^2,0)$. A lower value of MMD indicates that the two distributions are similar (yet negative values cannot be compared by magnitude). 
+
+As a test statistic, MMD can be used to perform a two-sample test to determine if the two distributions are significantly different. In this case, a value higher than some threshold (determined by the distribution of MMD under the null hypothesis) would indicate that the two distributions are significantly different. In the context of synthetic data evaluation, a low MMD value would indicate that the synthetic data is not unreasonably different.
+
+Reference:
+> Gretton, A., Borgwardt, K.M., Rasch, M.J., Smola, A., Schölkopf, B., & Smola, A. (2012). A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25), 723–773. [http://jmlr.org/papers/v13/gretton12a.html](http://jmlr.org/papers/v13/gretton12a.html)
+
+### Feature Importance Overlap (FIO)
+FIO is a generic metric that measures the overlap in top-k selected features between a model trained on real data and a model trained on synthetic data in predicting the target analysis variable. The metric is calculated as follows:
+
+$$
+\text{FIO}_k = \frac{|\text{Top-$k$ features from real data} \cap \text{Top-$k$ features from synthetic data}|}{k},
+$$
+
+In the actual implementation we select top-5%, 10%, 25%, and 50% of the features (where possible), and return all successful results. A higher value of FIO indicates that the synthetic data is a good representation of the real data in terms of feature selection. However, for the lowest top-k selections (e.g., 5% or 10%), it should be *very close* to 1 for a good synthetic data set, while for higher top-k selections (e.g., 25% or 50%) it can be more divergent and still indicate a good synthetic data set.