CRAN task view proposal: StatisticalSimulation

[maxwestphal]

Scope

The CTV will cover packages dedicated to statistical simulation studies, with particular focus on method comparison studies using simulated data. Included packages should provide several functions useful for this purpose. We will restrict the CTV to simulation studies based on tabular data with no restriction on study design (observational studies, randomized controlled trials, …), but not covering other types of data (e.g., images, audio, video, text, …). Included packages should be sufficiently generic and not focus only on a narrow domain (e.g., packages that only simulate infectious disease data, or packages with main focus on sample size estimation). Our planned structure for the CTV is as follows:

• Simulation study design and implementation
• Data generation
  • Univariate
  • Multivariate
  • Longitudinal
  • Time-to-event
  • Missing data
• Analysis and Reporting of simulation studies
  • Performance measures
  • Uncertainty quantification
  • Visualization

Hereby, we differentiate between tasks involved in conducting simulation studies, including “tooling” supporting design and implementation, methods that implement data generation processes and, lastly, methods for the analysis and reporting of simulation studies. We recognize that some packages will contribute methods for several of these tasks.
We will initially focus on parametric methods for data generation where we anticipate a large number of relevant packages. Important packages for nonparametric and semi-parametric data generation methods (e.g. resampling-based methods, plasmode simulation) will also be covered but with less detail than parametric approaches to limit the scope, at least initially. In the following, we formulate an explicit list of exclusion criteria.

Exclusion criteria

• High performance computing packages (has its own CTV), unless they have specific wrapper functions for facilitating simulation studies.
• Packages streamlining simulations with narrower scopes e.g. packages facilitating simulations only for infectious disease modelling, or sample size calculation/power analyses.
• Reproducible research tools (has its own CTV) that are not specific to simulation studies (i.e., to the execution or reporting of simulation study results).
• Packages focusing on the generation of non-tabular data (e.g., images, audio, video, text data)
• Packages with a primary purpose of sharing real datasets.

Packages

Aspect	Packages
Design and implementation	simFrame, simulator, simTool, faux, SimDesign, simsalapar, tidyMC, simhelpers, parsim
Data generation	modgo, simdata, SimJoint, simstudy, synthpop, SimCorrMix, bigsimr, PoisBinOrdNonNor, simPop, semiArtificial, distrSim, covsim, fabricatr
Analysis and Reporting	simTool, SimDesign, simMetric, rsimsum, simhelpers

Overlap

We expect overlap with existing CTVs. Whenever possible we would prefer linking to the existing CTV instead of listing a redundant collection of packages. For instance, we would certainly refer to the “Distributions” CTV in our “Data generation” section and to the “Reproducible Research” CTV in our “Analysis and Reporting” section. Because the CTV “HighPerformanceComputing” is very relevant for implementing statistical simulation studies, we will also certainly link to it. In all these cases, we think that it is reasonable to also mention a few specific packages from these CTVs if they have a very high relevance for statistical simulation studies.

Even though we will not list individual packages with functionality relevant to statistical simulation but only in a very narrow scope (see exclusion criteria), we could still refer to relevant CTVs which cover such packages such as “Agriculture”, “Pharmacokinetics”, or “Psychometrics”.

Maintainers

• Max Westphal
• Michael Kammer
• Edouard Bonneville
• Anne-Laure Boulesteix
• James Pustejovsky

[jpiaskowski]

Hello, thank for you this proposal. Do you have a draft of this somewhere we can view?

[maxwestphal]

Hello, thank for you this proposal. Do you have a draft of this somewhere we can view?

Thanks for this question @jpiaskowski. No, so far we have only prepared the proposal (as posted in this issue).

[zeileis]

Thanks, Max @maxwestphal et al., for the proposal. I like the idea, even if it is a bit narrow at the moment.

I think in addition to the given exclusion criteria, one would have to exclude tools for simulation-based inference (especially Bayesian MCMC).

Also, in the tools for "data generation" one should probably only include those packages that are substantial/complex enough and not just the usual "random numbers form certain distributions". The latter is part of the "Distributions" task view.

For possible inclusion: Would it make sense to include packages designed for benchmark studies and computer experiments (e.g., DiceDesign)?

I'm also tagging Matthias @matthias-da here because he might have useful comments as maintainer of simPop and active task view maintainer.

P.S.: CRAN package SimNPH would probably be added in the data generation section.

[matthias-da]

Dear Achim and all

I had a look at the new task view proposal on statistical simulation. Personally, I very much welcome it: it fills a real gap, the proposer team is strong, and the structure (design/implementation, data generation, analysis/reporting) matches how method-comparison studies are actually conducted. I have three structural concerns and a few smaller notes.

1. Two different things are currently mixed under “Data generation”. Packages like simstudy, SimCorrMix, covsim or bigsimr generate data from an analyst-specified data-generating process (DGP), so the ground truth is known by construction. Which is exactly what the stated scope (“method comparison studies”) requires for evaluating bias, coverage or RMSE.
   synthpop, simPop, and others do something different: they estimate models from a given real dataset to produce a synthetic replica. Their primary purposes are privacy-preserving data dissemination (synthpop, simPop), realistic mock/teaching data (modgo) and data augmentation (semiArtificial); the “truth” is data-conditional rather than user-specified, and such tools serve method comparison only via additional design steps. Listing them together also contradicts the proposal’s statement that nonparametric and semiparametric approaches will be covered “with less detail”. My suggestion would be to structure “Data generation” into (a) model-based generation with known truth, (b) resampling-based approaches, and (c) fully synthetic replicas of real data, with the data-type breakdown (multivariate, longitudinal, time-to-event, etc.) nested within these categories, or at least made explicit.

2. A related overlap point is that data synthesis is a core statistical disclosure control (SDC) topic and is already covered in the Official Statistics task view (e.g. sdcMicro, simPop, synthpop, MicSim). Following the proposal’s own policy on overlapping packages, a cross-link would be cleaner than flat inclusion. I’d be happy to coordinate this from the Official Statistics side.

3. The distinction between design-based and model-based simulation is currently missing from the scope. simFrame’s distinctive feature is design-based simulation, i.e. repeated sampling from a finite population under complex survey designs and missingness scenarios. In that paradigm, simPop has a principled role because it generates the synthetic finite population from which repeated samples constitute the known truth. Framed this way, simPop clearly belongs, but as a population generator for design-based studies rather than as a generic multivariate DGP tool.

4. Smaller notes:

• For missing-data simulation, missMethods and mice (via ampute) might be included. The MissingData CRAN Task View might also be added to the overlap/cross-link list.
• Under design and implementation, SimEngine seems a notable omission, as does DeclareDesign (fabricatr is already listed). causalSim and simDAG would cover DAG-based DGPs for causal method-comparison studies.
• faux is listed under design, but it is primarily a data-generation package.
• Since “simulation” is an overloaded term, the scope might explicitly exclude simulation modelling (e.g. discrete-event simulation, ODE-based simulation, agent-based and micro-simulation) as opposed to simulation studies of statistical methods. That would help prevent misfiled package suggestions later and could even be reflected in the task view’s title.

For transparency: I am the (co-)author of the simFrame and simPop paper and package, so please read points 1–2 with that in mind. But my recommendation is to clarify the conceptual distinctions.

Another person to involve (e.g., proofreading) might be Carolin Strobl from the University of Zurich. She recently published a book https://link.springer.com/book/10.1007/978-3-662-70561-2 on simulation studies in R.

Kind regards
Matthias

[zeileis]

Very helpful, Matthias @matthias-da, thanks for taking the time! To me this all sounds very reasonable. (And I had also thought about asking Caro for feedback as well...good point.)

[maxwestphal]

Thank you very much @zeileis and @matthias-da for these constructive and helpful comments!

We are happy to add the suggested (and so far missing packages); the list in our proposal was just a small initial selection. For your comments regarding scope, structure and overlap with other CTVs (like OfficialStatistics, which we missed so far) - I will discuss these aspects with the team and post a longer answer here in a couple of days.

[zeileis]

Ok, sounds good! 👍

Thanks for the quick feedback

[maxwestphal]

We discussed the feedback so far in the team and I will try to reply to all the comments by Achim @zeileis and Matthias @matthias-da below. Many thanks again for your valuable suggestions and questions. For most suggestions, we already came to an agreement and answered accordingly. For a few suggestions, we anticipate that a final choice can only be made while writing the CTV (with more information) but not necessarily right now in this early phase. I also included a revised version of the CTV proposal at the end of this post.

Regarding the comments by Achim

• We agree that the scope is quite narrow currently. But we would rather create a good CTV with a narrow scope in a reasonable timeframe (with the option for later extensions) than a lacking CTV with a scope that is (too) broad.
• We added an EC for "simulation-based inference".
• We added an EC for "only generating data from certain distributions" (probably the wording can still be improved)
• Regarding your question on packages for "benchmark studies" and "computer experiments". We will include packages in our CTV that are useful for statistical simulation studies, even if their primary target use case is different. DiceDesign appears to be such a package. On the other hand, we will not include packages which are useful for benchmark studies or computer experiments but not for statistical simulation studies (at least at the start, see first point).
• We added the SimNPH package to our package list.

Regarding the comments by Matthias

1. We agree that there are more dimensions to categorize data generating functions/packages than those mentioned in our first proposal version, including the example given here (model-based vs. resampling-based vs. synthesizers). This dimension will certainly be utilized in our CTV. However, we are not yet 100% sure on the following important questions and feel that it is hard to fully answer them now (while only having a small list of packages that will be included):
   a. How many categories there should be and how to define them? (The proposal by Matthias sounds like a good starting point.)
   b. How to label the categories?
   c. How to include this information in a useful way in the CTV structure?
2. We are currently in favor of including these packages in our CTV as well, mainly because it appears that (a) there are some additional packages we would like to cover and (b) the reason for inclusion/exclusion of individual packages in this category may be different here than it is for the SDC CTV.
3. We are happy to include the suggested packages simPop and simFrame and also differentiate between the types of data generation mentioned (design-based, model-based), potentially with different labels.
4. Minor comments
   a. Missing data: it looks all/most packages relevant for our context are covered in the MissingData CTV, so we will link to that (added as an EC).
   b. We added all mentioned packages to the example list in the revised proposal.
   c. A very good point. We have taken this as the incentive to change the CTV title to StatisticalSimulationStudies as this (at least for us) corresponds better to the more narrow scope that we intend. Also, we added the mentioned topics to the list of ECs.

Revised CTV proposal

(Achim @zeileis, please let me know if I should edit the issue title and/or initial post to reflect our changes instead of posting the revised version here. I am not sure if you have a standard/preference for this. Thanks.)

Revised CTV proposal

StatisticalSimulationStudies

Scope

The CTV will cover packages dedicated to statistical simulation studies, with particular focus on method comparison studies using simulated data. Included packages should provide several functions useful for this purpose. We will restrict the CTV to simulation studies based on tabular data with no restriction on study design (observational studies, randomized controlled trials, …), but not covering other types of data (e.g., images, audio, video, text, …). Included packages should be sufficiently generic and not focus only on a narrow domain (e.g., packages that only simulate infectious disease data, or packages with main focus on sample size estimation). Our planned structure for the CTV is as follows:

• Simulation study design and implementation
• Data generation
  • Univariate
  • Multivariate
  • Longitudinal
  • Time-to-event
  • Missing data
• Analysis and Reporting of simulation studies
  • Performance measures
  • Uncertainty quantification
  • Visualization

Hereby, we differentiate between tasks involved in conducting simulation studies, including “tooling” supporting design and implementation, methods that implement data generation processes and, lastly, methods for the analysis and reporting of simulation studies. We recognize that some packages will contribute methods for several of these tasks.
We will initially focus on parametric methods for data generation where we anticipate a large number of relevant packages. Important packages for nonparametric and semi-parametric data generation methods (e.g. resampling-based methods, plasmode simulation) will also be covered but with less detail than parametric approaches to limit the scope, at least initially. There will most likely be further attributes which will be used to categorize included packages, i.e. the structure mentioned above is only a starting point and will likely be iterated. In the following, we formulate an explicit list of exclusion criteria.

Exclusion criteria

• High performance computing packages (has its own CTV), unless they have specific wrapper functions for facilitating simulation studies.
• Packages streamlining simulations with narrower scopes e.g. packages facilitating simulations only for infectious disease modelling, or sample size calculation/power analyses.
• Reproducible research tools (has its own CTV) that are not specific to simulation studies (i.e., to the execution or reporting of simulation study results).
• Packages focusing on the generation of non-tabular data (e.g., images, audio, video, text data)
• Packages with a primary purpose of sharing real datasets.
• Packages focusing on simulation-based inference (e.g., Bayesian MCMC).
• Packages focusing of generating data from certain distributions (has its own CTV).
• Packages focusing on implementing mechanisms to realize missing data (has its own CTV).
• Packages focusing on discrete-event simulation
• Packages focusing on ODE-based simulation
• Packages focusing on agent-based and micro-simulation

Packages

Aspect	Packages
Design and implementation	simFrame, simulator, simTool, faux, SimDesign, simsalapar, tidyMC, simhelpers, parsim, DiceDesign, SimEngine, DeclareDesign
Data generation	modgo, simdata, SimJoint, simstudy, synthpop, SimCorrMix, bigsimr, PoisBinOrdNonNor, simPop, semiArtificial, distrSim, covsim, fabricatr, SimNPH, simPop, simFrame, causalSim, simDAG
Analysis and Reporting	simTool, SimDesign, simMetric, rsimsum, simhelpers

Overlap

We expect overlap with existing CTVs. Whenever possible we would prefer linking to the existing CTV instead of listing a redundant collection of packages. For instance, we would certainly refer to the “Distributions” CTV in our “Data generation” section and to the “Reproducible Research” CTV in our “Analysis and Reporting” section. Because the CTV “HighPerformanceComputing” is very relevant for implementing statistical simulation studies, we will also certainly link to it. In all these cases, we think that it is reasonable to also mention a few specific packages from these CTVs if they have a very high relevance for statistical simulation studies.

Even though we will not list individual packages with functionality relevant to statistical simulation but only in a very narrow scope (see exclusion criteria), we could still refer to relevant CTVs which cover such packages such as “Agriculture”, “Pharmacokinetics”, or “Psychometrics”.

Maintainers

• Max Westphal
• Michael Kammer
• Edouard Bonneville
• Anne-Laure Boulesteix
• James Pustejovsky

[zeileis]

Thanks for the follow-up, this looks good to me. What do you think, Matthias @matthias-da?

Regarding the title: "StatisticalSimulationStudies" is really long. So unless this is really always used as a compound (like "NaturalLanguageProcessing"), I would go either for "StatisticalSimulation" or "SimulationStudies" as they are shorter to type when using ctv::install.views() and friends. But if there is a clear case for the long three-word title, then this is feasible as well.

[matthias-da]

I am a bit lost to give further feedback before a draft is written.
But based on your response, I think you are on the right track.

@zeileis , @maxwestphal : I am happy to read and comment the CTV, when the CTV (and its sections/dimensions) is available as draft. Of course, I can try to give response when you have the feeling that another opinion is useful for any issue that pops up.

@maxwestphal : Minor remark. R package synthpop is also useful for synthetic (twin) data generation, i.e. when you include simPop, you should also think about my "competitors" ;-)

[zeileis]

OK, thanks, Matthias @matthias-da . I take this as a signal that we should proceed and then ask for more feedback when there is a full first draft.

Dirk @eddelbuettel, Julia @jpiaskowski, Nathalie @tuxette, Roger @rsbivand, what do you think? You can also just give a thumbs up if you endorse moving ahead.

[jpiaskowski]

I like the idea and would like to see a draft. @maxwestphal, is it worth mentioning in the CTV that many linear modelling packages provide simulation of model-based data (e.g. lme4)?

[zeileis]

Thanks for the endorsements, so we can move on to the next step.

Max @maxwestphal, please prepare a full draft. I would recommend that you set up your own GitHub project for this and already mimic the structure of official task views. Then when we are ready for publication you can transfer your project to us.

[maxwestphal]

Thanks for the endorsements, so we can move on to the next step.

Max @maxwestphal, please prepare a full draft. I would recommend that you set up your own GitHub project for this and already mimic the structure of official task views. Then when we are ready for publication you can transfer your project to us.

Many thanks Achim @zeileis and also the others for the additional helpful comments and endorsements. We will gladly start preparing a full draft version. I will follow-up here if (a) critical questions come up or (b) we have a first draft version that we can share with you.