Move calculation to Prediction class (#39)

* Begin moving calculation to Prediction class

* Move Brier calculation

* Refactor

* Cleanup

* Document changes

* Update documentation with new API

Author: yhoiseth

CHANGELOG.md

@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Calculation of scores for when the order of alternatives matters.
 - Design philosophy documentation.
 - Meta documentation
+- Simplify API
 
 ### Changed
 - Decimal input for probabilities instead of integer.

README.md

@@ -39,26 +39,28 @@ There are several ways to score predictions like these. Here, we are using [Brie
 
 (See [plot.py](plot.py) for the code that generated this chart.)
 
-Now, back to our election example. The following code scores the predictions.
+Now, back to our election example. The following code scores the predictions using the Brier scoring rule.
 
 ```python
 from decimal import Decimal
 
-from predictionscorer import calculators, predictions
+from predictionscorer import predictions
+
+true_alternative_index = 1 # Alternative 0 is Hillary Clinton. Alternative 1 is Donald Trump.
 
 george = predictions.Prediction(
-    probabilities=(Decimal(60), Decimal(40)) # George put Clinton at 60 % and Trump at 40 %.
-)
-kramer = predictions.Prediction(
-    probabilities=(Decimal(35), Decimal(65)) # Kramer put Clinton at 35 % and Trump at 65 %.
+    probabilities=(Decimal(60), Decimal(40)), # George put Clinton at 60 % and Trump at 40 %.
+    true_alternative_index=true_alternative_index,
 )
 
-brier = calculators.Brier(
-    true_alternative_index=1 # Alternative 0 is Hillary Clinton. Alternative 1 is Donald Trump.
+print(george.brier_score) # Decimal('0.72')
+
+kramer = predictions.Prediction(
+    probabilities=(Decimal(35), Decimal(65)), # Kramer put Clinton at 35 % and Trump at 65 %.
+    true_alternative_index=true_alternative_index,
 )
 
-print(brier.calculate(george)) # Decimal('0.72')
-print(brier.calculate(kramer)) # Decimal('0.245')
+print(kramer.brier_score) # Decimal('0.245')
 ```
 
 As you can see, Kramer’s score is _lower_ than George’s. How can a better prediction give a lower score? The thing is, with Brier scores, the lower, the better. To help your intuition, you can consider a Brier score as the _distance from the truth_. (A perfect prediction yields 0, while the worst possible prediction yields 2.)
@@ -70,21 +72,18 @@ The above example is binary — there are only two alternatives. But sometimes y
 ```python
 from decimal import Decimal
 
-from predictionscorer import calculators, predictions
+from predictionscorer import predictions
 
 prediction = predictions.Prediction(
     probabilities=(
         Decimal(55), # Clinton
         Decimal(35), # Trump
         Decimal(10), # Other
-    )
-)
-
-brier = calculators.Brier(
+    ),
     true_alternative_index=1,
 )
 
-print(brier.calculate(prediction)) # Decimal('0.735')
+print(prediction.brier_score) # Decimal('0.735')
 ```
 
 #### If the order matters
@@ -102,12 +101,12 @@ Sometimes, the ordering of alternatives matters. For example, consider the follo
 
 We [now know that the answer is 3,230.78](https://us.spindices.com/indices/equity/sp-500). This means that, among our alternatives, the one with index 1 turned out to be correct. But notice that index 2 is closer to the right answer than index 3. In such cases, the regular Brier score is a poor measure of forecasting accuracy. Instead, we can use [the ordered categorical scoring rule](https://goodjudgment.io/Training/Ordered_Categorical_Scoring_Rule.pdf).
 
-The code below should look familiar, except that we are now using the `OrderedCategorical` calculator instead of the `Brier` calculator.
+The code below should look familiar, except that we are now setting `order_matters=True`:
 
 ```python
 from decimal import Decimal
 
-from predictionscorer import calculators, predictions
+from predictionscorer import predictions
 
 prediction = predictions.Prediction(
     probabilities=(
@@ -116,13 +115,11 @@ prediction = predictions.Prediction(
         Decimal(30),
         Decimal(20),
     ),
-)
-
-ordered_categorical = calculators.OrderedCategorical(
     true_alternative_index=1,
+    order_matters=True,
 )
 
-print(ordered_categorical.calculate(prediction)) # Decimal('0.2350')
+print(prediction.brier_score) # Decimal('0.2350')
 ```
 
 ## Changelog

predictionscorer/calculators.py

@@ -2,17 +2,122 @@
 from decimal import Decimal
 
 
+class Base:
+    true_alternative_index: int
+
+    def __init__(self, true_alternative_index: int) -> None:
+        self.true_alternative_index = true_alternative_index
+
+
+class Brier(Base):
+    """
+    Calculates scores when the order of predictions does not matter.
+    """
+
+    def calculate(self, prediction: "Prediction") -> Decimal:
+        score = Decimal("0.00")
+        for index, probability in enumerate(prediction.probabilities):
+            first_term = probability / Decimal(100)
+            second_term = Decimal(
+                "1.00" if index == self.true_alternative_index else "0.00"
+            )
+            score = score + (first_term - second_term) ** 2
+        return score
+
+
+class OrderedCategorical(Base):
+    """
+    Calculates scores when the order of predictions matters.
+    """
+
+    def calculate(self, prediction: "Prediction") -> Decimal:
+        total = Decimal("0.00")
+        pair_count = self._pair_count(prediction.probabilities)
+        for index in range(pair_count):
+            pair = Prediction(
+                self._split_probabilities(index, prediction.probabilities),
+                true_alternative_index=1,
+            )
+            score = self._score_pair(index, pair)
+            total += score
+        return self._average(total, pair_count)
+
+    @staticmethod
+    def _pair_count(probabilities: typing.Tuple[Decimal, ...]) -> int:
+        """
+        We need one fewer pairs than the number of alternatives. For example, if there are three alternatives — A, B and C, the pairs are:
+
+        - A and BC
+        - AB and C
+        """
+        return len(probabilities) - 1
+
+    @staticmethod
+    def _average(total: Decimal, count: int) -> Decimal:
+        return total / Decimal(count)
+
+    def _score_pair(self, index: int, pair: "Prediction") -> Decimal:
+        assert len(pair.probabilities) == 2, "There must be exactly two probabilities."
+        true_alternative_index = 0 if index > self.true_alternative_index else 1
+        brier_calculator = Brier(true_alternative_index=true_alternative_index)
+        return brier_calculator.calculate(pair)
+
+    @staticmethod
+    def _split_probabilities(
+        index: int, probabilities: typing.Tuple[Decimal, ...]
+    ) -> typing.Tuple[Decimal, Decimal]:
+        """
+        Given an index and a tuple of more than two probabilities, return a pair of grouped probabilities.
+        """
+        assert len(probabilities) > 2
+        first_part = probabilities[: (index + 1)]
+        second_part = probabilities[(index + 1) :]
+        sum_first_part = Decimal(sum(first_part))
+        sum_second_part = Decimal(sum(second_part))
+        return sum_first_part, sum_second_part
+
+
 class Prediction:
     """
     This class encapsulates probabilities for a given question.
     """
 
+    _cached_brier_score: typing.Optional[Decimal] = None
+    order_matters: bool
     probabilities: typing.Tuple[Decimal, ...]
+    true_alternative_index: int
 
-    def __init__(self, probabilities: typing.Tuple[Decimal, ...]) -> None:
+    def __init__(
+        self,
+        probabilities: typing.Tuple[Decimal, ...],
+        true_alternative_index: int,
+        order_matters: bool = False,
+    ) -> None:
         """
         2 or more probabilities are required. Make sure that they sum to 100.
         """
-        assert len(probabilities) >= 2, "A prediction needs at least two probabilities."
         assert sum(probabilities) == 100, "Probabilities need to sum to 100."
+        length = len(probabilities)
+        assert length >= 2, "A prediction needs at least two probabilities."
+        assert (
+            true_alternative_index >= 0
+        ), "The true alternative index cannot be negative"
+        assert (
+            true_alternative_index <= length - 1
+        ), "Probabilities need to contain the true alternative"
+        self.order_matters = order_matters
         self.probabilities = probabilities
+        self.true_alternative_index = true_alternative_index
+
+    @property
+    def brier_score(self) -> Decimal:
+        if isinstance(self._cached_brier_score, Decimal):
+            return self._cached_brier_score
+        calculator: typing.Union[Brier, OrderedCategorical] = (
+            OrderedCategorical(true_alternative_index=self.true_alternative_index)
+            if self.order_matters
+            else Brier(true_alternative_index=self.true_alternative_index)
+        )
+        score = calculator.calculate(self)
+        self._cached_brier_score = score
+        return score