Merge pull request #1063 from mdekstrand/feature/866-accumulators

Rework the metric interface and collection to use explicit accumulators

Author: mdekstrand

docs/guide/GettingStarted.ipynb

@@ -34,7 +34,7 @@ For an example, let's start with importing things to run a quick batch:
     >>> from lenskit.batch import recommend
     >>> from lenskit.data import load_movielens
     >>> from lenskit.splitting import sample_users, SampleN
-    >>> from lenskit.metrics import RunAnalysis, RBP
+    >>> from lenskit.metrics import MeasurementCollector, RBP
 
 Load and split some data:
 
@@ -49,7 +49,7 @@ Configure and train the model:
 
 Generate recommendations:
 
-    >>> recs = recommend(pop_pipe, split.test.keys(), n_jobs=1)
+    >>> recs = recommend(pop_pipe, split.test.keys())
     >>> recs.to_df()
               user_id  item_id     score  rank
     0 ...                                    1
@@ -58,13 +58,11 @@ Generate recommendations:
 
 And measure their results:
 
-    >>> ra = RunAnalysis()
-    >>> ra.add_metric(RBP())
-    >>> scores = ra.measure(recs, split.test)
-    >>> scores.list_summary()    # doctest: +ELLIPSIS
-              mean    median     std
-    metric
-    RBP    0.06...   0.02... 0.07...
+    >>> collect = MeasurementCollector()
+    >>> collect.add_metric(RBP())
+    >>> collect.measure_collection(recs, split.test)
+    >>> collect.summary_metrics()    # doctest: +ELLIPSIS
+    {... 'RBP.mean': 0.06..., ...}
 
 
 The :py:func:`predict` function works similarly, but for rating predictions.

docs/guide/batch.rst

@@ -86,22 +86,20 @@ Reporting Metrics
 ~~~~~~~~~~~~~~~~~
 
 Reporting the metrics themselves is relatively straightforward.  The
-:py:meth:`lenskit.bulk.RunAnalysis.measure` method returns a results object
-contianing the metrics for individual lists, the global metrics, and easy access
-(through :meth:`~lenskit.bulk.RunAnalysis.list_summary`) to summary statistics
-of per-list metrics, optionally grouped by keys such as model name.
+:py:meth:`lenskit.eval.MeasurementCollector.list_metrics` and
+:py:meth:`lenskit.eval.MeasurementCollector.summary_metrics` methods return the
+individual list metrics or overall summaries for a set of lists.
 
 The following code will produce a table of algorithm scores for hit rate, NDCG
 and MRR, assuming that your algorithm identifier is in a column named ``model``
 and the target list length is in ``N``::
 
-    rla = RunAnalysis()
-    rla.add_metric(Hit(n=N))
-    rla.add_metric(NDCG(n=N))
-    rla.add_metric(RecipRank(n=N))
-    results = rla.measure(recs, test)
-    # group by agorithm
-    model_metrics = results.list_summary('model')
+    mc = MeasurementCollector()
+    mc.add_metric(Hit(n=N))
+    mc.add_metric(NDCG(n=N))
+    mc.add_metric(RecipRank(n=N))
+    mc.measure(recs, test)
+    mc.list_metrics()
 
 You can then use :py:meth:`pandas.DataFrame.to_latex` to convert ``algo_scores``
 to a LaTeX table to include in your paper.

docs/guide/documenting.rst

@@ -28,7 +28,7 @@ There are two ways to directly call a prediction accuracy metric:
 
 * Pass a single item list with scores and a ``rating`` field.
 
-For evaluation, you will usually want to use :class:`~lenskit.metrics.RunAnalysis`,
+For evaluation, you will usually want to use :class:`~lenskit.metrics.MeasurementCollector`,
 which takes care of calling the prediction metric for you.
 
 Missing Data

docs/guide/evaluation/predictions.rst

@@ -9,9 +9,9 @@ The :py:mod:`lenskit.metrics.ranking` module contains the core top-*N* ranking
 accuracy metrics (including rank-oblivious list metrics like precision, recall,
 and hit rate).
 
-Ranking metrics extend the :py:class:`RankingMetricBase` base class in addition
-to :py:class:`ListMetric` and/or :py:class:`GlobalMetric`, return a score given
-a recommendation list and a test rating list, both as :py:class:`item lists
+Ranking metrics extend the :py:class:`RankingMetricBase` base class, often in
+addition to :py:class:`ListMetric`, and return a score given a recommendation
+list and a test rating list, both as :py:class:`item lists
 <lenskit.data.ItemList>`; most metrics require the recommendation item list to
 be :py:attr:`~lenskit.data.ItemList.ordered`.
 

docs/guide/evaluation/rankings.rst

@@ -64,6 +64,11 @@ fall into the following categories:
   also directly supports “global” metrics that are computed over an entire run
   instead of one list at a time.
 
+  .. versionchanged:: 2026.1
+
+    ``RunAnalysis`` has been superseded by
+    :class:`~lenskit.metrics.MeasurementCollector`.
+
 .. important::
 
     The default options of some metrics, particularly

docs/guide/migrating.rst

@@ -42,7 +42,16 @@ Breaking Changes
   is now decomposed. All metrics based on listwise measurements or intermediate
   results should directly extend from ``Metric`` or :class:`~lenskit.metrics.ListMetric`.
   (:pr:`983`)
-- ``GlobalMetric`` no longer inherits from ``Metric``, and may be removed in a future release.
+- ``GlobalMetric`` has been removed. Code needing to compute global metrics
+  should directly compute over item lists.
+- :class:`~lenskit.metrics.MeasurementCollector` no longer supports defaults
+  when adding metrics.
+- :class:`~lenskit.metrics.RunAnalysis` is deprecated, and no longer supports
+  explicit defaults or global metrics.  Use
+  :class:`~lenskit.metrics.MeasurementCollector` instead.  Its global metric
+  columns have also changed in some cases.
+- :class:`~lenskit.metrics.ListGini` and :class:`~lenskit.metrics.ExposureGini`
+  now require an item vocabulary or dataset as input.
 - Stopped providing wheels for macOS on Intel.  Users who still need to run
   LensKit on Intel-based Macs should use the Conda packages (available in conda-forge and
   `prefix.dev`_).
@@ -86,6 +95,9 @@ Performance Changes
 Minor Changes
 -------------
 
+- :func:`lenskit.metrics.call_metric` has been renamed to
+  :func:`~lenskit.metrics.measure_list`, and the old name preserved as a
+  deprecated alias.
 - Pipeline type-checking for `ArrayLike` component inputs no longer works, due to
   a breaking change in NumPy 2.4.  No LensKit components used `ArrayLike` as an
   input or output data type.

docs/releases/2026.rst

@@ -0,0 +1,19 @@
+# This file is part of LensKit.
+# Copyright (C) 2018-2023 Boise State University.
+# Copyright (C) 2023-2026 Drexel University.
+# Licensed under the MIT license, see LICENSE.md for details.
+# SPDX-License-Identifier: MIT
+
+"""
+Data accumulation support
+"""
+
+from ._proto import Accumulator, AccumulatorFactory
+from ._value import ValueStatAccumulator, ValueStatistics
+
+__all__ = [
+    "Accumulator",
+    "AccumulatorFactory",
+    "ValueStatAccumulator",
+    "ValueStatistics",
+]

src/lenskit/data/accum/__init__.py

@@ -0,0 +1,27 @@
+# This file is part of LensKit.
+# Copyright (C) 2018-2023 Boise State University.
+# Copyright (C) 2023-2026 Drexel University.
+# Licensed under the MIT license, see LICENSE.md for details.
+# SPDX-License-Identifier: MIT
+
+from ._proto import Accumulator
+
+
+class ObjectListAccumulator[T](Accumulator[T, list[T]]):
+    """
+    An accumulator lists of objects.
+    """
+
+    values: list[T]
+
+    def __init__(self):
+        self.values = []
+
+    def __len__(self) -> int:
+        return len(self.values)
+
+    def add(self, value: T):
+        self.values.append(value)
+
+    def accumulate(self) -> list[T]:
+        return self.values

src/lenskit/data/accum/_object.py

@@ -0,0 +1,40 @@
+# This file is part of LensKit.
+# Copyright (C) 2018-2023 Boise State University.
+# Copyright (C) 2023-2026 Drexel University.
+# Licensed under the MIT license, see LICENSE.md for details.
+# SPDX-License-Identifier: MIT
+
+from __future__ import annotations
+
+from typing import Protocol, runtime_checkable
+
+
+@runtime_checkable
+class AccumulatorFactory[X, R](Protocol):
+    def create_accumulator(self) -> Accumulator[X, R]:
+        """
+        Create an accumulator for the results of this object.
+
+        Return:
+            An accumulator.
+        """
+        ...
+
+
+@runtime_checkable
+class Accumulator[X, R](Protocol):
+    """
+    Protocol implemented by data accumulators.
+    """
+
+    def add(self, value: X) -> None:
+        """
+        Add a single value to this accumulator.
+        """
+        ...
+
+    def accumulate(self) -> R:
+        """
+        Compute the accumulated value from this accumulator.
+        """
+        ...