QUESTION: how to deal with renv::restore() failing to instal packages from source

[stefanocoretta]

I am trying to figure out if I am doing something wrong.

I am on macOS Silicon and I frequently update the OS and command line tools. It happens often that I have to restore from the lock file and when I do there are a few packages that always fail to install because they are installed from source. Among the culprits, Matrix and mgcv.

Why is renv installing from source even though there must be binary packages available?

[kevinushey]

Among the culprits, Matrix and mgcv.

This likely implies you need to install the Fortran libraries; see e.g. https://mac.r-project.org/tools/.

Why is renv installing from source even though there must be binary packages available?

Are you sure? What versions of these packages are you using? Note that renv::restore() wants to install the exact recorded version of the package currently in the lockfile, and if it's older, then binaries may not be available.

[stefanocoretta]

I have Fortran. I just discovered that on macOS you need extra libraries (so it is not quite true that you just need XCode and Fortran) as explained in this well hidden page: https://mac.r-project.org/bin/

I'm not sure the binary is available actually, I just assumed older versions are always available as binaries? Or may only some OS/R package versions combinations are available as binaries?

So basically I would have to flag to users that wish to clone and use a repo with renv that they should install those extra libraries that are not in XCode and Fortran if they are on Mac? As I said, learning that you need XCode and Fortran is quite straightforward, but the bit on extra libraries and how to get them is not that obvious (I stumbled upon it yesterday from a comment, not even an answer, on stackexchange 😬).

I understand it is not quite a renv issue, but maybe flagging this in the renv documentation might help. I assume that on Linux one just has those libraries or can get them with sudo-apt and on Windows that's what RTools gives you (?), but on Mac it is quite hermetic.

[kevinushey]

For posterity, can you share the full command output you're seeing?

[stefanocoretta]

Sure!

Error: Error installing package 'Matrix': 
================================== 
* installing *source* package ‘Matrix’ ... 
** this is package ‘Matrix’ version ‘1.7-2’ 
** package ‘Matrix’ successfully unpacked and MD5 sums checked 
** using staged installation 
** libs 
using C compiler: ‘Apple clang version 17.0.0 (clang-1700.0.13.5)’ 
using SDK: ‘MacOSX15.5.sdk’ 
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"SuiteSparse/SuiteSparse_config" -I/opt/R/arm64/include -fvisibility=hidden -fPIC -falign-functions=64 -Wall -g -O2 -c Csparse.c -o Csparse.o 
In file included from Csparse.c:1: 
./Mdefines.h:41:11: fatal error: 'libintl.h' file not found 
  41 | # include <libintl.h> 
     | ^~~~~~~~~~~ 
1 error generated. 
make: *** [Csparse.o] Error 1 
ERROR: compilation failed for package ‘Matrix’ 
* removing ‘/Users/ste/repos/coretta2018itapol/renv/staging/1/Matrix’ install of package 'Matrix' failed [error code 1]

The error was fixed after installing gettext using the instructions here: https://mac.r-project.org/bin/. Specifically:

source("https://mac.R-project.org/bin/install.R")
install.libs("gettext")

But other packages will need other libs, so best to do:

source("https://mac.R-project.org/bin/install.R")
install.libs("all")

[kevinushey]

Ack, yes -- thanks for the heads up. Indeed, recent versions of macOS have removed the libintl headers, so it's now necessary to get those from some external source (e.g. gettext with Homebrew, or more officially using the instructions you found.)

I should really consider suggesting this as a solution to installation errors of this form on macOS.

Does mgcv installation fail with the same error, or did it just fail to install because Matrix failed to install?

[stefanocoretta]

I think mgcv failed for the same missing library, but I can't quite remember.

The other issue is those few packages that use OpenMP (like data.table). In theory one could set the packages to build without OpenMP but I was unsuccessful in trying that (this is yet a separate issue). I could successfully build data.table with OpenMP support but I had to set Makevars and that broke building mgcv. (Maybe I did something wrong, but it feels like you do need multiple manual steps... See https://forum.posit.co/t/building-packages-from-source-on-macos-silicon/210508).

I think there is no real single solution at least for macOS.