yardstick 1.4.0

Breaking Changes

• The global option yardstick.event_first (deprecated in 0.0.7) now throws an error. Use the event_level argument of individual metric functions instead. (#632)

• conf_mat() now throws an error if anything is passed to ... (deprecated in 1.0.0). This argument has had no effect since case weight support was added. (#632)

• roc_auc(), roc_aunp(), roc_aunu(), and roc_curve() now throw an error if a non-empty list is passed to options (deprecated in 1.0.0). Use the pROC package directly if you need these features. (#632)

Deprecations

• dots_to_estimate(), metric_summarizer(), and metric_vec_template() (soft-deprecated in 1.2.0) now warn for all users. See the yardstick 1.2.0 release notes for recommended replacements. (#632)

New Metrics

• gini_coef() computes the normalized Gini coefficient for regression, which measures ranking ability based on the Lorenz curve. (#147)

• mse() computes the mean squared error. (#560)

• rmse_relative() computes the relative root mean squared error, normalizing RMSE by the range of the true values. (#527)

• fall_out() and miss_rate() compute the false positive rate and false negative rate respectively. (#336)

• markedness() computes the markedness metric (PPV + NPV - 1), the predictive power analog of j_index(). (#27)

• roc_dist() computes the Euclidean distance from the (sensitivity, specificity) point to the ideal point (1, 1) in ROC space. (#148)

• sedi() computes the Symmetric Extremal Dependence Index, a prevalence-independent skill metric for binary classification that remains reliable at extreme class imbalance. (#630)

• ranked_prob_score() computes the ranked probability score for ordinal classification data. (#524)

• weighted_interval_score() is a new quantile metric. (#569)

Improvements

• Added checks to all metrics for the na_rm argument. (#349)

• Added improved argument checking for metrics with additional arguments. (#519)

• Added documentation pages for each metric type (e.g., ?class-metrics, ?numeric-metrics) listing all available metrics with their direction and range. (#547, #540)

• All class metrics and probability metrics now include mathematical formulas in their documentation. (#605)

• All metrics now document their valid range of output values. (#572)

• Documentation pages for metrics with alternate argument values now emphasize using metric_tweak() when building metric sets. (#626)

• Survival metrics performance has been improved. (#576)

• brier_class() has gained the event_level argument. (#515)

• get_metrics() has been added to return a metric_set() containing all metrics of a specified type. (#534)

• metric_set() now provides a more informative error message when estimate is not explicitly named for class/prob or survival metric sets. (#504)

• roc_curve() has gained a thresholds argument for specifying custom thresholds at which the curve is evaluated. (#488)

Bug Fixes

• brier_class() no longer returns NaN with extreme value case weights. (#614)

• classification_cost() documentation now correctly refers to the cost column of the costs data frame. (#343)

• mpe() documentation now includes the formula and clarifies the interpretation of positive and negative values. (#345)

• poisson_log_loss() now handles 0-valued estimates without returning Inf or NaN. (#513)

• Fixed a bug where ranked probability metrics didn't work in combination with other classification metrics in metric_set(). (#539)

• Fixed documentation to show equations correctly. (#541)

Developer

• Added infrastructure for quantile metrics. (#569)

• Added infrastructure for survival metrics on the linear predictor. (#551)

• new_metric() and related functions gain an optional range argument to store the valid output range of a metric. (#572)

• Removed crayon as a suggested package. (#574)

yardstick 1.3.2

• All messages, warnings and errors has been translated to use {cli} package (#517, #522).

yardstick 1.3.1

Bug Fixes

• Bug was fixed in roc_curve_survival() where wrong weights were used. (#495, @asb2111).

• Output of roc_curve_survival() now returns columns in same order as roc_curve(). (#498)

yardstick 1.3.0

New Metrics

• The Brier score for survival data was added with brier_survival().

• The Integrated Brier score for survival data was added with brier_survival_integrated().

• The Concordance index for survival data was added with concordance_survival().

• Time-Dependent ROC curves estimation for right-censored data can now be
  calculated with roc_curve_survival().

• Time-Dependent ROC AUC estimation for right-censored data can now be
  calculated with roc_auc_survival().

Improvements

• demographic_parity(), equalized_odds(), and equal_opportunity() are new metrics for measuring model fairness. Each is implemented with the new_groupwise_metric() constructor, a general interface for defining group-aware metrics that allows for quickly and flexibly defining fairness metrics with the problem context in mind.

• metric_set() can now be used with a combination of dynamic and static survival metrics.

• Added a print method for metrics and metric sets (#435).

• All warnings and errors have been updated to use the cli package for increased clarity and consistency. (#456, #457, #458)

• brier_survival_integrated() now throws an error if input data only includes 1 evalution time point. (#460)

• Clarifying documentation about how event_level always default to "first. (#432)

Bug Fixes

• Metrics now throw more informative error if estimate argument is wrongly used. (#443)

Breaking Changes

• Curve metrics now throw an informative error instead of returning NA when missing values are found and na_rm = FALSE. (#344)

yardstick 1.2.0

New Metrics

• The Brier score for classification was added with brier_class() (#139).

Improvements

• The global option, yardstick.event_first, has been hard deprecated in favor
  of using explicit argument, event_level. Setting this option will now
  produce an warning, but won't have any effect. (#173)

• Removed start-up message about event_level argument.

• yardstick metric functions now use a pure tidyselect interface for truth,
  estimate, and the ... of class probability metrics (#322).

• Changed the default aspect ratio for PR curves to be 1.0.

• Error messages now show what user-facing function was called (#348).

• classification and probability metrics now fully support class_pred objects
  from {probably} package (#341).

Bug Fixes

• Using metric_set() on a metric created with metric_tweak() will no longer
  produces an error, and will favor arguments set with metric_tweak() (#351).

• Metric summarizers no longer error if column names in data conflict with
  argument names (#382).

• conf_mat() no longer throw errors listed as internal (#327).

Developer

• metric_vec_template() is being soft deprecated in favor of a more manual
  and flexible metric creation approach. yardstick_remove_missing() and
  yardstick_any_missing() have been added for treatment of missing values.
  check_class_metric(), check_numeric_metric(),
  and check_prob_metric() have been added to perform standardized input
  checking for classification, regression and class probability metrics
  respectively. These changes mean that it is the developer's responsibility
  to perform validation of truth and estimate input. (#337).

• metric_summarizer() is being soft deprecated in favor of the more specific
  newly added class_metric_summarizer(), numeric_metric_summarizer(),
  prob_metric_summarizer(), and curve_metric_summarizer() (#322).

• dots_to_estimate() is being soft deprecated along with
  metric_summarizer(). dots_to_estimate() is not needed with
  prob_metric_summarizer(), and curve_metric_summarizer() (#329).

yardstick 1.1.0

• Emil Hvitfeldt is now the maintainer (#315).

• Improved on the chained error thrown by metric_set() when one of the metric
  computations fails (#313).

yardstick 1.0.0

• All yardstick metrics now support case weights through the new case_weights
  argument. This also includes metric-adjacent functions like roc_curve(),
  pr_curve(), conf_mat(), and metric_set().

• The options argument of roc_curve(), roc_auc(), roc_aunp(),
  roc_aunu(), and metrics() that was passed along to the pROC package is
  now deprecated and no longer has any affect. This is a result of changing to
  an ROC curve implementation that supports case weights, but does not support
  any of the previous options. If you need these options, we suggest wrapping
  pROC yourself in a custom metric (#296).

• conf_mat() now ignores any inputs passed through ... and warns if you
  try to do such a thing. Previously, those were passed on to base::table(),
  but with the addition of case weight support, table() is no longer used
  (#295).

• Fixed a small mistake in ccc() where the unbiased covariance wasn't being
  used when bias = FALSE.

• j_index() now throws a more correct warning if 0 is in the denominator
  when computing sens() internally. Additionally, in the multiclass case it
  now removes the levels where this occurs from the multiclass weighted average
  computation, which is consistent with how other metrics were updated to handle
  this in #118 (#265).

• Improved on some possible ambiguity in the documentation of the data
  argument for all metrics (#255).

• purrr has been removed from Suggests.

• The pROC package has been removed as a dependency (#300).

• Moved the Custom Metrics vignette to tidymodels.org (#236).

yardstick 0.0.9

• New metric poisson_log_loss() was added (#146).

• sensitivity() and specificity() now work correctly with the tune and
  workflowsets packages (#232).

• roc_curve() now throws a more informative error if truth doesn't have any
  control or event observations.

• dplyr 1.0.0 is now required. This allowed us to remove multiple usages of
  dplyr::do() in favor of dplyr::summarise(), which can now return packed
  data frame columns and multiple rows per group.

• Removed internal hardcoding of "dplyr_error" to avoid issues with an
  upcoming dplyr 1.0.8 release (#244).

• Updated test suite to testthat 3e (#243).

• Internal upkeep has been done to move from rlang::warn(.subclass = ) to
  rlang::warn(class = ), since the .subclass argument has been deprecated
  (#225).

yardstick 0.0.8

• New metric_tweak() for adjusting the default values of optional arguments in
  an existing yardstick metric. This is useful to quickly adjust the defaults
  of a metric that will be included in a metric_set(), especially if that
  metric set is going to be used for tuning with the tune package (#206, #182).

• New classification_cost() metric for computing the cost of a poor class
  probability prediction using user-defined costs (#3).

• New msd() for computing the mean signed deviation (also called mean
  signed difference, or mean signed error) (#183, @datenzauberai).

• class_pred objects from the probably
  package are now supported, and are automatically converted to factors before
  computing any metric. Note that this means that any equivocal values are
  materialized as NA (#198).

• The kap() metric has a new weighting argument to apply linear or
  quadratic weightings before computing the kappa value (#2, #125, @jonthegeek).

• When sens() is undefined when computing ppv(), npv(), j_index(), or
  bal_accuracy(), a sensitivity warning is now correctly thrown, rather than
  a recall warning (#101).

• The autoplot() method for gain curves now plots the curve line
  on top of the shaded polygon, resulting in a sharper look for the
  line itself (#192, @eddjberry).

• The autoplot() methods for conf_mat now respect user-defined dimension
  names added through conf_mat(dnn = ) or from converting a table with
  dimension names to a conf_mat (#191).

• Added an as_tibble() method for metric_set objects. Printing a
  metric_set now uses this to print out a tibble rather than a data frame
  (#186).

• Re-licensed package from GPL-2 to MIT. See consent from copyright holders
  here (#204).

yardstick 0.0.7

• The global option, yardstick.event_first, has been deprecated in favor of
  the new explicit argument, event_level. All metric functions that previously
  supported changing the "event" level have gained this new argument.
  The global option was a historical design decision that can be classified as
  a case of a hidden argument.
  Existing code that relied on this global option will continue to work in this
  version of yardstick, however you will now get a once-per-session warning
  that requests that you update to instead use the explicit event_level
  argument. The global option will be completely removed in a future version.
  To update, follow the guide below (#163).

  `options(yardstick.event_first = TRUE)`  -> `event_level = "first"` (the default)
  `options(yardstick.event_first = FALSE)` -> `event_level = "second"`
  

• The roc_auc() Hand-Till multiclass estimator will now ignore levels in
  truth that occur zero times in the actual data. With other methods of
  multiclass averaging, this usually returns an NA, however, ignoring
  levels in this manner is more consistent with implementations in the
  HandTill2001 and pROC packages (#123).

• roc_auc() and roc_curve() now set direction = "<" when computing the
  ROC curve using pROC::roc(). Results were being computed incorrectly with
  direction = "auto" when most probability values were predicting the wrong
  class (#123).

• mn_log_loss() now respects the (deprecated) global option
  yardstick.event_first. However, you should instead change the relevant
  event level through the event_level argument.

• metric_set() now strips the package name when auto-labeling functions
  (@rorynolan, #151).

• There are three new helper functions for more easily creating custom
  metric functions: new_class_metric(), new_prob_metric(), and
  new_numeric_metric().

• Rcpp has been removed as a direct dependency.