Skip to content

Sampling tests to strict #150

Open
@rstub

Description

@rstub

It can happen that the difference in the confidence intervals is larger than the 0.005 you are expecting:

  library(fwildclusterboot)
  set.seed(123123)
  dqrng::dqRNGkind("Xoroshiro128+")
  dqrng::dqset.seed(123123)
  
  data1 <<- fwildclusterboot:::create_data(
    N = 1000,
    N_G1 = 20,
    icc1 = 0.5,
    N_G2 = 20,
    icc2 = 0.2,
    numb_fe1 = 10,
    numb_fe2 = 10,
    #seed = 908361239,
    seed = 123123,
    weights = 1:N / N
  )
  
  lm_fit <- lm(proposition_vote ~ treatment + log_income,
               data = data1
  )
  
  boot1 <- boottest(lm_fit,
                    param = c("log_income"),
                    clustid = c("group_id2"),
                    B = 99999,
                    sampling = "dqrng"
  )
#> Warning: Please note that the seeding behavior for random number generation for
#> `boottest()` has changed with `fwildclusterboot` version 0.13.
#> 
#> It will no longer be possible to exactly reproduce results produced by versions
#> lower than 0.13.
#> 
#> If your prior results were produced under sufficiently many bootstrap
#> iterations, none of your conclusions will change.  For more details about this
#> change, please read the notes in
#> [news.md](https://cran.r-project.org/web/packages/fwildclusterboot/news/news.html).
#> This warning is displayed once per session.
#> Too guarantee reproducibility, don't forget to set a global random seed
#> **both** via `set.seed()` and `dqrng::dqset.seed()`.
#> This message is displayed once every 8 hours.
  
  boot2 <- boottest(lm_fit,
                    param = c("log_income"),
                    clustid = c("group_id2"),
                    B = 99999,
                    sampling = "standard"
  )
  all.equal(confint(boot1), confint(boot2))
#> [1] "Mean relative difference: 0.005275242"

Created on 2024-05-13 with reprex v2.1.0

Running the the two boottest() repeatedly w/o fixing the seed I see differences up to 0.007. This probably explains the surprising results I got in daqana/dqrng#80 when doing reverse dependency checks for the new version of dqrng.

Could you upload a version of fwildclusterboot that only changes the the expectation w.r.t. the confidence interval?

  expect_equal(confint(boot1), confint(boot2), tolerance = 0.0075)

Note: I am basing my analysis on v0.13, which I think corresponds to the current CRAN version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions