Develop a vignette that explains and demonstrates how to use the benchmarking and validation framework. This will help in interpreting the model metrics compared to the 'true' error model at the end.
The vignette should:
- Start with the processed model outputs and observations expected by the framework.
- Explain schemas, including units and ensemble representation.
- Explain preparation including temporal aggregation and alignment.
- Walk through validation functions
- Explain how and why each metric is used, and which metrics use ensemble mean vs the full ensemble.
- Generate and interpret plots, metrics, and validation report.
- Explain how users can substitute their own processed data.
Simulate observations so that the expected errors are known and can be compared to actual metrics. Include the equations and code used to generate them as a side note, but keep the vignette focused on using the validation framework.
Inputs and simulated observations
The vignette should explain how the simulated observations were generated from an error model with known parameterizations.
Here is a statistical model of simulated observations:
$$
o_{v,t} = m_{v,t} + S_v(\beta_v + \epsilon_{v,t}),
\qquad
\epsilon_{v,t} \sim N(0,\tau_v^2)
$$
Where
-
$o_{v,t}$ is the simulated observation;
-
$_v$ indexes model output variable;
-
$t$ indexes the observation timestep;
-
$m_{v,t}$ is ensemble mean after aggregation;
-
$S_v$ scales the error to output $v$;
-
$\beta_v$ is relative bias; and
-
$\tau_v$ controls random error.
$o_{v,t}$ can be computed thus:
set.seed(3)
output_scale <- stats::sd(ensemble_mean)
# Assume bias = 30% of SD; error = 50% of SD
relative_bias <- 0.3
relative_error <- 0.5
simulated_observation <- ensemble_mean +
output_scale * (
relative_bias +
stats::rnorm(
length(ensemble_mean),
mean = 0,
sd = relative_error
)
)This code can be used to generate the observation fixture, but these observations should be precomputed so that they are easy to inspect as a text file.
Develop a vignette that explains and demonstrates how to use the benchmarking and validation framework. This will help in interpreting the model metrics compared to the 'true' error model at the end.
The vignette should:
Simulate observations so that the expected errors are known and can be compared to actual metrics. Include the equations and code used to generate them as a side note, but keep the vignette focused on using the validation framework.
Inputs and simulated observations
The vignette should explain how the simulated observations were generated from an error model with known parameterizations.
Here is a statistical model of simulated observations:
Where
This code can be used to generate the observation fixture, but these observations should be precomputed so that they are easy to inspect as a text file.