Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
367b408
add theory documentation
sallen12 Dec 24, 2024
af0a183
Merge branch 'main' into docu
frazane Dec 24, 2024
f05bc95
add readthedocs config
frazane Jan 16, 2025
d39db2b
add readthedocs files
frazane Jan 16, 2025
84d4c0f
delete obsolete files
frazane Jan 16, 2025
e409203
update gitignore
frazane Jan 16, 2025
c5179af
update docs requirements
frazane Jan 16, 2025
18b8859
use docs requirements file
frazane Jan 16, 2025
d47f514
update docs requirements
frazane Jan 16, 2025
b1eaf7a
update docs requirements
frazane Jan 16, 2025
af0ec14
misc fixes and test cross-references
frazane Jan 17, 2025
74d0c90
remove old file from index
frazane Jan 17, 2025
71bc671
install the local package code for the build
frazane Jan 17, 2025
141da57
expand API reference documentation
frazane Jan 17, 2025
0a61e5f
delete mkdocs config file
frazane Jan 18, 2025
6b33a4b
wip on crps docstrings
frazane Feb 1, 2025
f248be4
more wip on crps docstrings
frazane Feb 1, 2025
ca58cea
pass doctest for examples in _crps.py
frazane Feb 1, 2025
b7740f2
more work on docstrings - math blocks, args, formatting, etc.
frazane Feb 1, 2025
66563af
add examples and pass doctest for variogram scores
frazane Feb 1, 2025
80fd35d
Merge branch 'main' into docu
frazane Mar 26, 2025
2c1b2f6
comment out latex math hook
sallen12 Apr 4, 2025
e6b0cf9
update theory docs
sallen12 Apr 4, 2025
516ff25
add documentation page for ensemble forecasts
sallen12 Apr 4, 2025
6f6a08d
add documentation page for ensemble forecasts
sallen12 Apr 4, 2025
cf5f95e
add documentation page for weighted scoring rules
sallen12 Apr 4, 2025
5eb6656
update docu index
sallen12 Apr 4, 2025
b922b16
Merge branch 'docu' of https://github.com/frazane/scoringrules into docu
sallen12 Apr 4, 2025
7f4603f
update axis naming args (#95)
nicholasloveday Apr 7, 2025
2e37351
fix m_axis bug
sallen12 Apr 14, 2025
40fad39
change test_kernels to use m_axis argument
sallen12 Apr 14, 2025
a79024f
fix m_axis bug in logs_ensemble
sallen12 Apr 14, 2025
c181cf3
increase N in crps ensemble tests
sallen12 Apr 14, 2025
ade193e
add exception in crps ensemble tests for akr approximations
sallen12 Apr 14, 2025
1db9052
change energy score function name
sallen12 Apr 14, 2025
e919a36
change variogram score function name
sallen12 Apr 14, 2025
fb905e9
add default threshold arguments to twcrps_ensemble
sallen12 Apr 14, 2025
c05fe6b
add default threshold arguments to owcrps_ensemble
sallen12 Apr 14, 2025
39f8eb0
add default threshold arguments to vrcrps_ensemble
sallen12 Apr 14, 2025
5b9b1cf
reduce tolerance on owgksuv_ensemble test
sallen12 Apr 14, 2025
e7b161e
add default threshold argument to weighted gksuv scores
sallen12 Apr 14, 2025
1ed244a
update default w_func in vrcrps_ensemble
sallen12 Apr 14, 2025
8442fc8
update test_wcrps
sallen12 Apr 14, 2025
9d8efef
reduce tolerance of owgks tests
sallen12 Apr 15, 2025
0c209f9
reduce tolerance of owgks tests
sallen12 Apr 15, 2025
ca483b5
update documentation link and prepare release tag
frazane May 29, 2025
4d91ec2
add deprecated aliases
frazane May 29, 2025
8db42c4
more minor fixes
frazane May 30, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ instance/
# Sphinx documentation
docs/_build/
docs/build/
_build*

# PyBuilder
.pybuilder/
Expand Down Expand Up @@ -151,3 +152,5 @@ scratch_nbs/
_devlog/
tests/output
.devcontainer/
docs/generated
scoringrules/vendored
10 changes: 5 additions & 5 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ repos:
hooks:
- id: codespell

- repo: https://github.com/frazane/check-latex-math
rev: v0.2.2
hooks:
- id: latex-math-validation
args: ["scoringrules/", "docs/", "*.py", "*.md"]
#- repo: https://github.com/frazane/check-latex-math
# rev: v0.2.2
# hooks:
# - id: latex-math-validation
# args: ["scoringrules/", "docs/", "*.py", "*.md"]
12 changes: 6 additions & 6 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ build:
tools:
python: "3.13"

# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs.yaml
# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py

# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
# python:
# install:
# - requirements: docs/requirements.txt
python:
install:
- requirements: docs/requirements.txt
53 changes: 33 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,38 @@
<!-- centered images -->
<p align="center">
<img src="docs/assets/images/banner_light.svg#gh-light-mode-only" />
<img src="docs/assets/images/banner_dark.svg#gh-dark-mode-only" />
<img src="docs/_static/banner_dark_long.svg#gh-dark-mode-only" alt="dark banner">
<img src="docs/_static/banner_light_long.svg#gh-light-mode-only" alt="light banner">
</p>

<br>
# Probabilistic forecast evaluation

[![CI](https://github.com/frazane/scoringrules/actions/workflows/ci.yaml/badge.svg?branch=main)](https://github.com/frazane/scoringrules/actions/workflows/ci.yaml)
[![codecov](https://codecov.io/github/frazane/scoringrules/graph/badge.svg?token=J194X4HEBH)](https://codecov.io/github/frazane/scoringrules)
[![pypi](https://img.shields.io/pypi/v/scoringrules.svg?colorB=<brightgreen>)](https://pypi.python.org/pypi/scoringrules/)

Scoringrules is a python library for evaluating probabilistic forecasts by computing
scoring rules and other diagnostic quantities. It aims to assist forecasting practitioners by
providing a set of tools based the scientific literature and via its didactic approach.
`scoringrules` is a python library that provides scoring rules to evaluate probabilistic forecasts.
It's original goal was to reproduce the functionality of the R package
[`scoringRules`](https://cran.r-project.org/web/packages/scoringRules/index.html) in python,
thereby allowing forecasting practitioners working in python to enjoy the same tools as those
working in R. The methods implemented in `scoringrules` are therefore based around those
available in `scoringRules`, which are rooted in the scientific literature on probabilistic forecasting.

The scoring rules available in `scoringrules` include, but are not limited to, the
- Brier Score
- Logarithmic Score
- (Discrete) Ranked Probability Score
- Continuous Ranked Probability Score (CRPS)
- Dawid-Sebastiani Score
- Energy Score
- Variogram Score
- Gaussian Kernel Score
- Threshold-Weighted CRPS
- Threshold-Weighted Energy Score


## Features

- **Fast** computations of several probabilistic univariate and multivariate verification metrics
- **Fast** computation of several probabilistic univariate and multivariate verification metrics
- **Multiple backends**: support for numpy (accelerated with numba), jax, pytorch and tensorflow
- **Didactic approach** to probabilistic forecast evaluation through clear code and documentation

Expand All @@ -28,7 +45,7 @@ pip install scoringrules

## Documentation

Learn more about scoringrules in its official documentation at https://frazane.github.io/scoringrules/.
Learn more about `scoringrules` in its official documentation at https://scoringrules.readthedocs.io/en/latest/.


## Quick example
Expand All @@ -41,26 +58,22 @@ fct = obs[:,None] + np.random.randn(100, 21) * 0.1
sr.crps_ensemble(obs, fct)
```

## Metrics
- Brier Score
- Continuous Ranked Probability Score (CRPS)
- Logarithmic score
- Error Spread Score
- Energy Score
- Variogram Score


## Citation
If you found this library useful for your own research, consider citing:
If you found this library useful, consider citing:

```
@software{zanetta_scoringrules_2024,
author = {Francesco Zanetta and Sam Allen},
title = {Scoringrules: a python library for probabilistic forecast evaluation},
title = {scoringrules: a python library for probabilistic forecast evaluation},
year = {2024},
url = {https://github.com/frazane/scoringrules}
}
```

## Acknowledgements
[scoringRules](https://cran.r-project.org/web/packages/scoringRules/index.html) served as a reference for this library. The authors did an outstanding work which greatly facilitated ours. The implementation of the ensemble-based metrics as jit-compiled numpy generalized `ufuncs` was first proposed in [properscoring](https://github.com/properscoring/properscoring), released under Apache License, Version 2.0.
- The widely-used R package [`scoringRules`](https://cran.r-project.org/web/packages/scoringRules/index.html)
served as a reference for this library, which greatly facilitated our work. We are additionally
grateful for fruitful discussions with the authors.
- The quality of this library has also benefited a lot from discussions with (and contributions from)
Nick Loveday and Tennessee Leeuwenburg, whose python library [`scores`](https://github.com/nci/scores)
similarly provides a comprehensive collection of forecast evaluation methods.
20 changes: 20 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
20 changes: 20 additions & 0 deletions docs/_static/banner_dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions docs/_static/banner_dark_long.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 20 additions & 0 deletions docs/_static/banner_light.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions docs/_static/banner_light_long.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/favicon.ico
Binary file not shown.
15 changes: 15 additions & 0 deletions docs/_static/logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
62 changes: 62 additions & 0 deletions docs/_templates/module.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{{ objname | escape | underline}}

.. automodule:: {{ fullname }}
:members:

{% block attributes %}
{% if attributes %}
.. rubric:: {{ _('Module Attributes') }}

.. autosummary::
{% for item in attributes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

{% block functions %}
{% if functions %}
.. rubric:: {{ _('Functions') }}

.. autosummary::
{% for item in functions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

{% block classes %}
{% if classes %}
.. rubric:: {{ _('Classes') }}

.. autosummary::
{% for item in classes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

{% block exceptions %}
{% if exceptions %}
.. rubric:: {{ _('Exceptions') }}

.. autosummary::
{% for item in exceptions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

{% block modules %}
{% if modules %}
.. rubric:: Modules

.. autosummary::
:toctree:
:recursive:
{% for item in modules %}
{{ item }}
{%- endfor %}

{% endif %}
{% endblock %}
9 changes: 0 additions & 9 deletions docs/api/brier.md

This file was deleted.

Loading
Loading