Error file not found after st_warp()

[dominicroye]

I have a 4 GB geotiff which I am reading as stars_proxy, and then I wanted to change to a lower resolution with st_wrap(). The thing is that, in some way, the temporary file is lost.

  rs <- read_stars(str_glue("{m}_norm_WD1.tiff"), proxy = T)
  rs <- st_warp(rs, crs = 4326, cellsize = .7, use_gdal = T, method = "average")  
  rs <- st_set_dimensions(rs, 3, values = time(tmp), names = "time")
  names(rs) <- "var_norm"
plot(rs)
downsample set to 8
trying to read file: /tmp/RtmppxO2d0/file113e7ea91a.tif

SesionInfo:
`
Linking to GEOS 3.12.1, GDAL 3.8.4, PROJ 9.3.1; sf_use_s2() is TRUE

sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] furrr_0.3.1 future_1.33.2 tictoc_1.2.1 stars_0.6-6 abind_1.4-5 lubridate_1.9.3 forcats_1.0.0 stringr_1.5.1 dplyr_1.1.4
[10] purrr_1.0.2 readr_2.1.5 tidyr_1.3.1 tibble_3.2.1 ggplot2_3.5.1 tidyverse_2.0.0 fs_1.6.4 sf_1.0-16 terra_1.7-78

loaded via a namespace (and not attached):
[1] s2_1.1.7 utf8_1.2.4 generics_0.1.3 lwgeom_0.2-14 class_7.3-22 KernSmooth_2.23-24 stringi_1.8.4 listenv_0.9.1
[9] digest_0.6.36 hms_1.1.3 magrittr_2.0.3 grid_4.4.1 timechange_0.3.0 e1071_1.7-14 DBI_1.2.3 fansi_1.0.6
[17] scales_1.3.0 codetools_0.2-20 cli_3.6.3 rlang_1.1.4 units_0.8-5 parallelly_1.37.1 munsell_0.5.1 withr_3.0.0
[25] tools_4.4.1 parallel_4.4.1 tzdb_0.4.0 colorspace_2.1-0 globals_0.16.3 vctrs_0.6.5 R6_2.5.1 proxy_0.4-27
[33] lifecycle_1.0.4 classInt_0.4-10 pkgconfig_2.0.3 pillar_1.9.0 gtable_0.3.5 glue_1.7.0 Rcpp_1.0.13 tidyselect_1.2.1
[41] rstudioapi_0.16.0 wk_0.9.2 compiler_4.4.1
`

[dominicroye]

To add some more information, when I use the argument debug = T it seems to work. The temporary file isn't deleted.

[edzer]

That's a clue. Could you check whether this fixed it?

[dominicroye]

Yes, it worked. Another question: if I read as stars_proxy, I was surprised that st_wrap() isn't adding the changes as lazy operations. Is this normal?

[edzer]

if I read as stars_proxy, I was surprised that st_wrap() isn't adding the changes as lazy operations. Is this normal?

I never gave it a thought - stars_proxy was implemented after stars, and the proxy functionality was added driven by use case. The main reason to use proxy is (i) to be able to work with downsampled images, and (ii) to process large grids full resolution out of memory (chunk-wise). For (i) I'm not sure if a warped downsampled raster is the same as a downampled warped raster; for (ii) you need to do the full raster anyway, as st_warp(use_gdal=TRUE) now does.

I can imagine it makes sense to handle stars_proxy separately if you first crop a proxy raster, then warp the cropped area; right now you'd first have to st_as_stars() the crop.

What's your use case?

[dominicroye]

I have to prepare several spatial-temporal NC files before using them with another method. The steps are to crop to a smaller extension with st_crop(), normalize the variable with st_apply(), and adjust the resolution from 0.25 to 0.7 using st_wrap().

[edzer]

Does the path crop -> st_as_stars -> warp work?

[dominicroye]

Yes! Thank you

[tylerhoecker]

I just wanted to provide a note here that I am attempting a similar workflow as the OP, and ran into this issue with geotiffs with >4000 bands. st_warp failed, in various ways, until I provided a reference raster created from a bounding box of my initial reference raster and failed when there were >2000 bands until I included debug = TRUE. Grateful I finally stumbled across this fix!