pkddown::build_site can't find Python when vignette depends on reticulate and Python

[ttimbers]

I am creating a vignette for the flair package (see this PR) and I can successfully knit the vignette on my laptop (the vignette depends on Python, reticulate and pandas), however when I try pkddown::build_site() I get the following error indicating that the Python specified in RETICULATE_PYTHON doesn't exist. This is not the case, as I routinely use Python via reticulate in RStudio on this machine, and as I stated above I can I can successfully knit the vignette on my laptop.

Error message:

── Building articles ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Reading 'vignettes/how_to_flair.Rmd'
Reading 'vignettes/python_flair.Rmd'
Error : Failed to render RMarkdown
* Quitting from lines 30-31 (python_flair.Rmd) 
* Error in py_discover_config(required_module, use_environment) : 
*   Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//RtmpxrALJF/callr-spr-b12b189abcf7) does not exist
In addition: Warning message:
The vignette title specified in \VignetteIndexEntry{} is different from the title in the YAML metadata. The former is "how_to_flair", and the latter is "Add some flair to your source code". If that is intentional, you may set options(rmarkdown.html_vignette.check_title = FALSE) to suppress this check. 
Error: callr subprocess failed: Failed to render RMarkdown
* Quitting from lines 30-31 (python_flair.Rmd) 
* Error in py_discover_config(required_module, use_environment) : 
*   Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//RtmpxrALJF/callr-spr-b12b189abcf7) does not exist
Type .Last.error.trace to see where the error occured

Here's the output of py_config():

python:         /opt/miniconda3/bin/python
libpython:      /opt/miniconda3/lib/libpython3.7m.dylib
pythonhome:     /opt/miniconda3:/opt/miniconda3
version:        3.7.6 (default, Jan  8 2020, 13:42:34)  [Clang 4.0.1 (tags/RELEASE_401/final)]
numpy:          /opt/miniconda3/lib/python3.7/site-packages/numpy
numpy_version:  1.18.4

NOTE: Python version was forced by RETICULATE_PYTHON

Note - I set RETICULATE_PYTHON in .Renviron via RETICULATE_PYTHON="/opt/miniconda3/bin"

[jayhesselberth]

Can you try running pkgdown::build_site(new_process = FALSE)?

[ttimbers]

Sadly, that doesn't fix it:

── Building articles ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Reading 'vignettes/how_to_flair.Rmd'
Reading 'vignettes/python_flair.Rmd'
Error: Failed to render RMarkdown
* Quitting from lines 30-31 (python_flair.Rmd) 
* Error in py_discover_config(required_module, use_environment) : 
*   Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//Rtmpuq9HB4/callr-spr-b0e151614435) does not exist
Run `rlang::last_error()` to see where the error occurred.
In addition: Warning message:
The vignette title specified in \VignetteIndexEntry{} is different from the title in the YAML metadata. The former is "how_to_flair", and the latter is "Add some flair to your source code". If that is intentional, you may set options(rmarkdown.html_vignette.check_title = FALSE) to suppress this check. 

[ttimbers]

I can trace the error to the pkgdown:::render_rmarkdown(...) function:

> build_articles()
── Building articles ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Reading 'vignettes/python_flair.Rmd'
Error: Failed to render RMarkdown
* Quitting from lines 30-31 (python_flair.Rmd) 
* Error in py_discover_config(required_module, use_environment) : 
*   Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//RtmpZXnZPV/callr-spr-b7404d4d4eb) does not exist
Run `rlang::last_error()` to see where the error occurred.
> rlang::last_error()
<error/rlang_error>
Failed to render RMarkdown
* Quitting from lines 30-31 (python_flair.Rmd) 
* Error in py_discover_config(required_module, use_environment) : 
*   Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//RtmpZXnZPV/callr-spr-b7404d4d4eb) does not exist
Backtrace:
 1. pkgdown::build_articles()
 2. purrr::walk(...)
 3. purrr::map(.x, .f, ...)
 4. pkgdown:::.f(.x[[i]], ...)
 5. pkgdown:::render_rmarkdown(...)
Run `rlang::last_trace()` to see the full context.

If I try build_articles(new_process = FALSE) (as I am really trying to do is build the vignettes) I get this error:

Error in build_articles(new_process = FALSE) : 
  unused argument (new_process = FALSE)

Which makes sense if you look at the first comment of the file containing the pkgdown:::render_rmarkdown(...) function:

#' Render RMarkdown document in a fresh session

So, to make this work, I guess there would need to be a way to state that pkgdown:::render_rmarkdown(...) should not be run in a new process, or have the new process somehow take an argument as to where RETICULATE_PYTHON is?

[cderv]

I think the error message gives us a hint

Python specified in RETICULATE_PYTHON (/opt/miniconda3/binR_PROFILE=/var/folders/65/4844hh0x68x64r5qwzc302tw0000gp/T//Rtmpuq9HB4/callr-spr-b0e151614435) does not exist

It seems that there is an issue with how environment variable is processed in callr (called by render_rmarkdown) from the one in .Renviron (I guess a project level file ?). Here, Two environment variables are mixed up, RETICULATE_PYTHON and R_PROFILE. I think the second is added by callr itself and there is something happening there...

One wild guess : in your .Renviron, is RETICULATE_PYTHON the last variable in the file ? Is there a newline after, at the end of the file ?

I think a minimal reprex can be done with callr on this one. I’ll try when I have access to a 💻

Hope it helps.

[cderv]

One wild guess : in your .Renviron, is RETICULATE_PYTHON the last variable in the file ? Is there a newline after, at the end of the file ?

It seems to be the root cause. Here is a reprex with callr only. I think you need to be sure to add a newline character at the end of your last line. This is often the case (like in yaml). as you'll see below, function like writeLines will do that by default.

What happens is I think that callr will append another environment to the one in your .Renviron but it will not add a newline before so it will append directely after the last one value. Not what you want !

Here is the reprex.

dir.create(temp_wd <- tempfile())

# with no newline character at the end
withr::with_dir(
  temp_wd, {
    cat("RETICULATE_PYTHON=/opt/miniconda3/bin", file = ".Renviron")
    callr::r_safe(function() {
      Sys.getenv("RETICULATE_PYTHON")
    })
  }
)
#> [1] "/opt/miniconda3/binR_PROFILE=C:\\Users\\chris\\AppData\\Local\\Temp\\RtmpwTt5aF\\callr-spr-1f2048656be8"

# with newline character at the end
withr::with_dir(
  temp_wd, {
    cat("RETICULATE_PYTHON=/opt/miniconda3/bin\n", file = ".Renviron")
    callr::r_safe(function() {
      Sys.getenv("RETICULATE_PYTHON")
    })
  }
)
#> [1] "/opt/miniconda3/bin"

# writeLines adds the newline character
withr::with_dir(
  temp_wd, {
    writeLines("RETICULATE_PYTHON=/opt/miniconda3/bin", con = ".Renviron")
    callr::r_safe(function() {
      Sys.getenv("RETICULATE_PYTHON")
    })
  }
)
#> [1] "/opt/miniconda3/bin"

unlink(temp_wd, recursive = TRUE)

^{Created on 2020-07-31 by the reprex package (v0.3.0.9001)}

I let you try.

[ttimbers]

Well I'll be... That was the problem. I will be more careful about adding the line break to these files in future. Thank-you very much for your help on this and apologies for the extra work this caused you folks!

[cderv]

No worries ! Always interesting to dig into those.
However, I wonder if this should be consider something that callr should take care of. Like checking for a newline character before appending.
@jayhesselberth what do you think ? Should this be reported in callr?