Cannot compile brms model with cmdstanr on Windows 10

[qdread]

I apologize if this duplicates an existing issue but it seems like there are many similar issues involving Windows. I am responsible for teaching Stan to federal employees and I am having trouble getting brms + CmdStan + cmdstanr combination to work on Windows 10 machines that they are required to use.

We experienced issues with cmdstanr 0.6.0 so we rolled back to 0.5.3, however we are still unable to compile any model with brm() via cmdstanr backend. Unfortunately, I do not have access to a machine with the configuration that the employees will be using so I have had to go back and forth with an IT person at another location trying to debug this issue. Here I am providing the error message we see on trying to compile the model, plus as much system information as I could get. Any help would be greatly appreciated! Thank you for this amazing package. It makes my life much better!

Compilation error

library(brms)
options(brms.backend = 'cmdstanr')
brm(mpg~hp, data=mtcars)

Error message:

Compiling Stan program...
INFO: Could not find files for the given pattern(s).

INFO: Could not find files for the given pattern(s).

p
rocess_begin: CreateProcess(C:\Windows\System32\cmd.exe, cmd /C "cscript /nologo /E:jscript ../tbb_2020.3/build/detect.js  /arch gcc", ...) failed.
../tbb_2020.3/build/windows.inc:29: pipe: No error

p
rocess_begin: CreateProcess(C:\Windows\System32\cmd.exe, cmd /C "cscript /nologo /E:jscript ../tbb_2020.3/build/detect.js  /runtime gcc", ...) failed.
../tbb_2020.3/build/windows.inc:33: pipe: No error
../tbb_2020.3/build/common.inc:81: *** Architecture not detected.  Stop.

mingw32-make: *** [stan/lib/stan_math/make/libraries:173: stan/lib/stan_math/lib/tbb/tbbmalloc.def] Error 2

Error: An error occured during compilation! See the message above for more information.

System information

> cmdstanr::check_cmdstan_toolchain()
The C++ toolchain required for CmdStan is setup properly!
>> Sys.which("mingw32-make")
mingw32-make 
          ""
> Sys.which("g++")
                                         g++ 
"C:\\rtools43\\X86_64~1.POS\\bin\\G__~1.EXE" 
>

Session info call:

> sessionInfo()
R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8 
[2] LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/Chicago
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] brms_2.19.0    Rcpp_1.0.11    cmdstanr_0.5.3

loaded via a namespace (and not attached):
[1] tidyselect_1.2.0     dplyr_1.1.2          farver_2.1.1        
 [4] loo_2.6.0            fastmap_1.1.1        tensorA_0.36.2      
 [7] shinystan_2.6.0      promises_1.2.0.1     shinyjs_2.1.0       
[10] digest_0.6.33        mime_0.12            lifecycle_1.0.3     
[13] StanHeaders_2.26.27  ellipsis_0.3.2       processx_3.8.2      
[16] magrittr_2.0.3       posterior_1.4.1      compiler_4.3.1      
[19] rlang_1.1.1          tools_4.3.1          igraph_1.5.0.1      
[22] utf8_1.2.3           knitr_1.43           prettyunits_1.1.1   
[25] bridgesampling_1.1-2 htmlwidgets_1.6.2    pkgbuild_1.4.2      
[28] plyr_1.8.8           dygraphs_1.1.1.6     abind_1.4-5         
[31] miniUI_0.1.1.1       withr_2.5.0          grid_4.3.1          
[34] stats4_4.3.1         fansi_1.0.4          xts_0.13.1          
[37] xtable_1.8-4         colorspace_2.1-0     inline_0.3.19       
[40] ggplot2_3.4.2        scales_1.2.1         gtools_3.9.4        
[43] cli_3.6.1            mvtnorm_1.2-2        crayon_1.5.2        
[46] generics_0.1.3       RcppParallel_5.1.7   reshape2_1.4.4      
[49] rstan_2.21.8         stringr_1.5.0        shinythemes_1.2.0   
[52] bayesplot_1.10.0     parallel_4.3.1       matrixStats_1.0.0   
[55] base64enc_0.1-3      vctrs_0.6.3          Matrix_1.5-4.1      
[58] jsonlite_1.8.7       callr_3.7.3          crosstalk_1.2.0     
[61] glue_1.6.2           codetools_0.2-19     ps_1.7.5            
[64] DT_0.28              distributional_0.3.2 stringi_1.7.12      
[67] gtable_0.3.3         later_1.3.1          munsell_0.5.0       
[70] tibble_3.2.1         colourpicker_1.2.0   pillar_1.9.0        
[73] htmltools_0.5.5      Brobdingnag_1.2-9    R6_2.5.1            
[76] shiny_1.7.4.1        lattice_0.21-8       markdown_1.7        
[79] backports_1.4.1      threejs_0.3.3        httpuv_1.6.11       
[82] rstantools_2.3.1.1   coda_0.19-4          gridExtra_2.3       
[85] nlme_3.1-162         checkmate_2.2.0      xfun_0.39           
[88] zoo_1.8-12           pkgconfig_2.0.3     

[jgabry]

however we are still unable to compile any model with brm() via cmdstanr backend

Sorry for the trouble! Do you know if it's possible to compile other models with cmdstanr that don't use brms? Based on the error message I'm guessing this would happen even when running e.g. cmdstanr::cmdstanr_example() and doesn't actually have to do with using brms.

Hopefully @rok-cesnovar @andrjohns or someone else who knows windows better than I do will have an idea here.

[jgabry]

Also @qdread, do you know if these same computers were able to previously compile models with older versions of cmdstanr and/or cmdstan? It's possible to install older versions of cmdstan itself with cmdstanr::install_cmdstan() using the version or release_url arguments. So if older versions of cmdstan used to work that's a possibility until we can sort out what's going on here.

[SteveBronder]

Windows 10 machines that they are required to use.

Tbc is this multiple machines that have had the same issue?

From here it looks like tbb is trying to detect the architecture of the PC and failing at that point

@ahartikainen pointed out on slack the following make file from tbb which suggests you can add environment variables to set the architecture yourself and avoid the windows issues

https://github.com/stan-dev/math/blob/34881d4b8b4d6bea963d2d69e2d12da8f723fb64/lib/tbb_2020.3/build/common.inc

[qdread]

@jgabry thanks for the responses. I know for sure that in February 2023, I taught the class and the students basically did a clean install of brms, cmdstan, and cmdstanr on the latest release of R available at the time. Whatever releases were current at that time worked. But I taught the class again in late June 2023. Different students this time, so again they installed brms, cmdstan, and cmdstanr from scratch on machines that were configured in probably the same way as in February. That is when the errors occurred. So maybe I will try to roll everything back to the versions that were current in Feb '23 to see if it works. If it does not work, maybe the next hypothesis would be that a security patch pushed down by agency IT sometime between February and June is not interacting well with cmdstan. We shall see ...

[jgabry]

Definitely keep us posted. I'm guessing you won't be the only one to run into issues like this and it would be good to know what worked.

[qdread]

Hi, I apologize for not following up on this. I have been continuing to troubleshoot but still getting the Architecture not detected error I posted originally. To respond to the queries from @jgabry:

• Most recent releases of cmdstanr, cmdstan, and brms are being used.
• The problem occurs with cmdstanr::cmdstanr_example() so it is not a brms issue.
• I rolled back cmdstanr to version 2.31.0 without success. In fact, the Architecture not detected error occurs at installation when trying to install 2.31.0, instead of first cropping up at compile time.

This is the relevant bit of sessionInfo() of the user that I am trying to help configure cmdstanr.

R version 4.3.2 (2023-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Thanks for any help you can provide. I would really like to figure out how to get this working. And one final question: is there any reason that any of the installation steps would not work properly if the user is not an admin? The users I am dealing with do not have admin privileges on their machines.

[qdread]

Apologies for the multiple pings, is there any way that someone can help reproduce this error and try to diagnose?

[andrjohns]

If the build/installation error is coming from the TBB, you can instead set cmdstan to use the libraries and headers from the RcppParallel package:

library(cmdstanr)

cmdstan_make_local(cpp_options = list(
  TBB_INC=system.file("include", package = "RcppParallel", mustWork = TRUE),
  TBB_LIB=system.file("lib", package = "RcppParallel", mustWork = TRUE)
))
cmdstan_rebuild()

As for issues with non-admins, the primary problem encountered is that some users are restricted from running arbitrary executables (like models compiled by cmdstan): https://discourse.mc-stan.org/t/problem-with-mcmc-execution-sample-of-cmdstanr-and-admin-rights/27913/2

[qdread]

@andrjohns Those are both great things to know, thank you! I will try the rebuild solution you provided, and also try to see if things work when elevating users to admin. I'll get back to you when I can meet with a user and test those things out (might be after New Year's).

[qdread]

Hi @andrjohns I believe I've isolated the problem, and it is related to users not having admin privileges. Is there any way to configure cmdstanr so that the compiled executables are written to a location where they can be run without admin privileges? If not, is there a way to temporarily elevate users to admin and make changes so that the executables can be run without being in admin mode? (Edit: I noticed on the link to the stan forums you put in your last post, that it may be possible for IT to change permissions to make it work, but could you please be more specific about what permissions need to be changed, if you can? thanks!)

[andrjohns]

You can change the directory where executables are stored using the output_dir argument.

As for the admin privileges, it can differ by organisation. Some organisations will only allow a executing programs on a specified whitelist, while others can be impacted by a particular Windows Defender setting.

Unfortunately if you have admin issues with executables, rstan is the only option for now

[andrjohns]

I've suggested a fix for this in #918, but it's likely to be a month or two before I've got time to work on it unfortunately.