Tests fail, vroom_fwf() doesn't leak a connection when opening fails

[nunotexbsd]

FreeBSD 16
R-4.5.2

<snip>
* checking installed files from 'inst/doc' ... OK
* checking files in 'vignettes' ... OK
* checking examples ... OK
* checking for unstated dependencies in 'tests' ... OK
* checking tests ...
  Running 'spelling.R'
  Running 'testthat.R'
 ERROR
Running the tests in 'tests/testthat.R' failed.
Last 13 lines of output:
      x
   1. +-testthat::expect_error(...) at test-connection.R:120:3
   2. | \-testthat:::expect_condition_matching_(...)
   3. |   \-testthat:::quasi_capture(...)
   4. |     +-testthat (local) .capture(...)
   5. |     | \-base::withCallingHandlers(...)
   6. |     \-rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
   7. \-testthat::expect_warning(...)
  -- Failure ('test-connection.R:120:3'): vroom_fwf() doesn't leak a connection when opening fails (permission denied) --
  Expected `expect_warning(...)` to throw a error.

  [ FAIL 2 | WARN 2 | SKIP 45 | PASS 1115 ]
  Error:
  ! Test failures.
  Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes ... OK
* checking running R code from vignettes ...
  'benchmarks.Rmd' using 'UTF-8'... OK
  'vroom.Rmd' using 'UTF-8'... OK
 OK
* checking re-building of vignette outputs ... SKIPPED
* DONE

Status: 1 ERROR, 1 NOTE
See
  '/wrkdirs/usr/ports/textproc/R-cran-vroom/work/vroom.Rcheck/00check.log'
for details.

Full log:

00check.log

[jennybc]

Can you show the full test output? Such as what you get from devtools::test().

[nunotexbsd]

Using

options(cli.num_colors = 0, cli.unicode = FALSE)
sink("tests.txt")
devtools::test(reporter = "summary")
sink()

If a better way to do this, please tell me.

big-int: ................................
chr: ..
col_types: ....�[32m.�[39m.........
connection: ...........12...
datetime: ............................................................................................................................................................................................................S.....................................
dbl: ........................
factor: ..........................................................................................
int: ........................
locale: ........................
logical: ............................
multi-byte: ........
multi-file: ..................
num: ................
path: ..................SS...S.......
problems: ......................................................................................
select: ..................
vroom_fwf: ........................................................................................
vroom_lines: .................
vroom_write: ....................................................
vroom: ............................................................................................................................................................................................................................................................................................................................................................

== Skipped =========================================================================================================================================================================================================
1. unambiguous times with and without daylight savings (test-datetime.R:309:3) - Reason: Not working on CI

2. can write to a zip file if the archive package is available (test-path.R:120:3) - Reason: {archive} is not installed

3. can write to a tar.gz file if the archive package is available (test-path.R:135:3) - Reason: {archive} is not installed

4. can read/write a compressed file with non-ascii characters in path (test-path.R:186:3) - Reason: {archive} is not installed

== Failed ==========================================================================================================================================================================================================
-- 1. Failure (test-connection.R:120:3): vroom_fwf() doesn't leak a connection when opening fails (permission denied) ----------------------------------------------------------------------------------------------
Expected `vroom_fwf(file(tfile), fwf_widths(c(6, 6, 6)), show_col_types = FALSE)` to throw a warning.
Backtrace:
    x
 1. +-testthat::expect_error(...) at test-connection.R:120:3
 2. | \-testthat:::expect_condition_matching_(...)
 3. |   \-testthat:::quasi_capture(...)
 4. |     +-testthat (local) .capture(...)
 5. |     | \-base::withCallingHandlers(...)
 6. |     \-rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 7. \-testthat::expect_warning(...)

-- 2. Failure (test-connection.R:120:3): vroom_fwf() doesn't leak a connection when opening fails (permission denied) ----------------------------------------------------------------------------------------------
Expected `expect_warning(...)` to throw a error.

== DONE ============================================================================================================================================================================================================

[jennybc]

Basically, the failing test is prepared to see a warning and an error and for some reason those aren't being thrown in your environment.

I've not seen any problem with this test on any of our CI platforms or nor on those tested by CRAN (CRAN's results are slowly updating for the new vroom release, but so far so good).

Are you an R user (vs. just looking after R stuff on some platform)? It would be interesting to walk through the failing test interactively in your environment to see what's happening.

[jennybc]

Are you running checks as root? That would explain why the lack of permissions does not, in fact, create any warning or error upon reading.

[nunotexbsd]

@jennybc yes, I'm running it as root...

My background is that I'm maintainer of vroom port at FreeBSD ( https://www.freshports.org/textproc/R-cran-vroom/ ) and we use poudriere package builder and tester. I usually run tests inside poudriere jail and I need su to get write permition.

Running it as unpreviledged poudriere user 'nobody' tests run OK.
I will try to figure a way of "fixing" this problem on port.

Thanks

root@16amd64-main:/wrkdirs/usr/ports/textproc/R-cran-vroom/work/vroom # su -m nobody -c "Rscript -e 'devtools::test()'"
ℹ Testing vroom
✔ | F W  S  OK | Context
✔ |         32 | big-int
✔ |          2 | chr
✔ |         14 | col_types
✔ |         15 | connection
✔ |      1 241 | datetime [1.5s]
✔ |         24 | dbl
✔ |         90 | factor
✔ |         24 | int
✔ |         24 | locale
✔ |         28 | logical
✔ |          8 | multi-byte
✔ |         18 | multi-file [2.1s]
✔ |         16 | num
✔ |      3  28 | path
✔ |         86 | problems
✔ |         18 | select
⠦ |         67 | vroom_fwf
Rows: 0 Columns: 3
-- Column specification --------------------------------------------------------

chr (3): a, b, c

i Use `spec()` to retrieve the full column specification for this data.
i Specify the column types or set `show_col_types = FALSE` to quiet this message.
✔ |         88 | vroom_fwf
✔ |         17 | vroom_lines
✔ |         52 | vroom_write
✔ |        348 | vroom [2.2s]

══ Results ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 12.3 s

── Skipped tests (4) ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• {archive} is not installed (3): test-path.R:120:3, test-path.R:135:3, test-path.R:186:3
• Not working on CI (1): test-datetime.R:309:3

[ FAIL 0 | WARN 0 | SKIP 4 | PASS 1173 ]
Warning messages:
1: In file(con, "w") :
  cannot open file '/wrkdirs/usr/ports/textproc/R-cran-vroom/work/vroom/compile_commands.json': Permission denied
2: cannot open the connection

[nunotexbsd]

@jennybc

Thanks for your help finding the problem.

[jennybc]

I'm going to reopen this because I do think I can make this test less fiddly and more true to its intent. But I'm glad it's no longer a blocker for you.

[nunotexbsd]

Nice!

For the time being I'm using a small workaround for FreeBSD port:

# Tests are supposed to be run with unprivileged user. Running as root, tests fail.
# See also: https://github.com/tidyverse/vroom/issues/611
do-test:
        @${MKDIR} ${WRKDIR}/tmp
        @${CHMOD} 777 ${WRKDIR}/tmp
        @cd ${WRKSRC} && \
                ${SETENV} ${TEST_ENV} TMPDIR=${WRKDIR}/tmp _R_CHECK_FORCE_SUGGESTS_=false \
                su -m nobody -c "${LOCALBASE}/bin/R CMD check --no-manual --no-build-vignettes ."

Thanks

[jennybc]

To replicate the problem I did this:

sudo Rscript -e 'devtools::test(filter = "connection")'

and indeed see:

Failure (test-connection.R:120:3): vroom_fwf() doesn't leak a connection when opening fails (permission denied)
Expected `vroom_fwf(file(tfile), fwf_widths(c(6, 6, 6)), show_col_types = FALSE)` to throw a warning.
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-connection.R:120:3
 2. │ └─testthat:::expect_condition_matching_(...)
 3. │   └─testthat:::quasi_capture(...)
 4. │     ├─testthat (local) .capture(...)
 5. │     │ └─base::withCallingHandlers(...)
 6. │     └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 7. └─testthat::expect_warning(...)

Failure (test-connection.R:120:3): vroom_fwf() doesn't leak a connection when opening fails (permission denied)
Expected `expect_warning(...)` to throw a error.

And after the fix:

~/work/vroom % sudo Rscript -e 'devtools::test(filter = "connection")'
ℹ Testing vroom
✔ | F W  S  OK | Context
✔ |         14 | connection

══ Results ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 14 ]