Something wrong with the 1.4.5 Mac binary on CRAN with Redshift

[ilarischeinin]

I discovered an odd issue, which I thought I'd report and can help troubleshoot if needed. The tl;dr is that I cannot connect to one Redshift server with the Mac binary from CRAN, but can if I compile from source.

The current Mac binary of RPostgres (1.4.5) seems to have an issue with some, but not all, Redshift servers. The command I'm running is

DBI::dbConnect(RPostgres::Redshift(), ...) # same result with RPostgres::Postgres()

and the error I get is:

connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

The error message is the same also when I don't provide a valid username/password, so it seems to happen before authentication.

The weird thing is that I get this error with only one Redshift server (our production one) and not with another (our staging) – even though they're both running the same version. Also, I can connect to both with psql from my Mac, and can connect to both with RPostgres from within a docker container running on my Mac. This made me suspect it is somehow related to the Mac version of RPostgres. And in fact, if instead of installing the binary from CRAN I compile it from source with (I have libpq installed with Homebrew):

install.packages("RPostgres", type = "source")

The problem goes away and I can connect to both servers.

I don't know if the solution is to simply recompile the Mac binary version that is available on CRAN, and that will fix it. If not, I can help troubleshoot on my end, in case that helps. The fact that this seems to affect some, but not all, Redshift servers can understandably make this hard the reproduce. Is there something I can do to see debugging messages from RPostgres/DBI?

My sessionInfo():

R version 4.2.3 (2023-03-15)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur ... 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] bit_4.0.5       compiler_4.2.3  cli_3.6.1       hms_1.1.3      
 [5] DBI_1.1.3       Rcpp_1.0.11     bit64_4.0.5     vctrs_0.6.3    
 [9] blob_1.2.4      lifecycle_1.0.3 pkgconfig_2.0.3 rlang_1.1.1    
[13] RPostgres_1.4.5

[krlmlr]

Thanks. Logging must be enabled when compiling, this is not an option here. This is really weird.

We could try harder to build the package so that fails to connect to Redshift the same way the CRAN build fails. What do you see with the following?

DBI::dbGetInfo(RPostgres::Postgres())
#> $driver.version
#> [1] '1.4.5'
#> 
#> $client.version
#> [1] '14.5'

^{Created on 2023-09-01 with reprex v2.0.2}

I'm a bit lost here. Logging the network communication and looking for differences? But this looks like serious overkill.

@s-u: Any idea how to troubleshoot libpq issues that happen (only) with the CRAN build?

[s-u]

You can check with the same binary from https://mac.r-project.org/bin/. If this is a bug in libpq then you can suggest an update to https://github.com/R-macos/recipes/blob/master/recipes/libpq

[ilarischeinin]

I get the same versions as you @krlmlr :

DBI::dbGetInfo(RPostgres::Postgres())
#> $driver.version
#> [1] '1.4.5'
#> 
#> $client.version
#> [1] '14.5'

^{Created on 2023-09-04 with reprex v2.0.2}

You can check with the same binary from https://mac.r-project.org/bin/.

It doesn't seem to have RPostgres available?

source("https://mac.R-project.org/bin/install.R")
#> 
#>  Usage: install.libs(names)
#> 
#>  Example: install.libs('cairo')
#> 
#>  names can be a vector or a special value 'all'.
#>  See args(install.libs) for defaults.
install.libs("RPostgres")
#> Downloading https://mac.R-project.org/bin/REPOS ...
#> Using repository  https://mac.R-project.org/bin/darwin20/x86_64 ...
#> Downloading index  https://mac.R-project.org/bin/darwin20/x86_64/PACKAGES ...
#> Error in deps(pkgs, db): Following binaries have no download candidates: RPostgres

^{Created on 2023-09-04 with reprex v2.0.2}

If this is a bug in libpq then you can suggest an update to https://github.com/R-macos/recipes/blob/master/recipes/libpq

Happy to, if that's what I should do. But to be honest I have no idea where the problem is. From the recipe you linked:

#: we only pick libpq from the entire PostgeSQL installation...

This makes me think that I do have both libpq and popstgresql installed with Homebrew, and I wonder if it could somehow be possible that in some scenario (our production Redshift) it used one, and in another scenario (our staging Redshift) the other. FWIW, here's output for brew info:

$ brew info libpq
==> libpq: stable 15.4 (bottled) [keg-only]
Postgres C API library
https://www.postgresql.org/docs/15/libpq.html
/usr/local/Cellar/libpq/15.4 (2,370 files, 28.2MB)
  Poured from bottle using the formulae.brew.sh API on 2023-08-30 at 12:31:07
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/lib/libpq.rb
License: PostgreSQL
==> Dependencies
Required: krb5 ✔, openssl@3 ✔
==> Caveats
libpq is keg-only, which means it was not symlinked into /usr/local,
because conflicts with postgres formula.

If you need to have libpq first in your PATH, run:
  echo 'export PATH="/usr/local/opt/libpq/bin:$PATH"' >> ~/.zshrc

For compilers to find libpq you may need to set:
  export LDFLAGS="-L/usr/local/opt/libpq/lib"
  export CPPFLAGS="-I/usr/local/opt/libpq/include"
==> Analytics
install: 72,927 (30 days), 184,647 (90 days), 309,228 (365 days)
install-on-request: 38,695 (30 days), 100,106 (90 days), 156,159 (365 days)
build-error: 9 (30 days)



$ brew info postgresql
Warning: Formula postgresql was renamed to postgresql@14.
==> postgresql@14: stable 14.9 (bottled)
Object-relational database system
https://www.postgresql.org/
/usr/local/Cellar/postgresql@14/14.9 (3,316 files, 44.6MB) *
  Poured from bottle on 2023-09-04 at 18:54:09
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/postgresql@14.rb
License: PostgreSQL
==> Dependencies
Build: pkg-config ✘
Required: icu4c ✔, krb5 ✔, lz4 ✔, openssl@3 ✔, readline ✔
==> Caveats
This formula has created a default database cluster with:
  initdb --locale=C -E UTF-8 /usr/local/var/postgres
For more details, read:
  https://www.postgresql.org/docs/14/app-initdb.html

To start postgresql@14 now and restart at login:
  brew services start postgresql@14
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/postgresql@14/bin/postgres -D /usr/local/var/postgres
==> Analytics
install: 55,121 (30 days), 164,673 (90 days), 268,824 (365 days)
install-on-request: 53,771 (30 days), 160,989 (90 days), 262,370 (365 days)
build-error: 6 (30 days)

[ilarischeinin]

I did some more testing with libpq and postgresql, and found out that:

• If I have both installed and compile RPostgres from source, it uses postgresql -> works
• If I have only libpq, it uses that -> works
• if I have only postgresql, It uses that -> works
• If I have neither, compiling RPostgres from source naturally fails

Here are the compilation outputs, in case they might be useful:

# with postgresql
install.packages("RPostgres", type = "source")
#> trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.5.tar.gz'
#> Content type 'application/x-gzip' length 1473682 bytes (1.4 MB)
#> ==================================================
#> downloaded 1.4 MB
#> 
#> * installing *source* package ‘RPostgres’ ...
#> ** package ‘RPostgres’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> Found pg_config includedir and libdir (PostgreSQL 14.9 (Homebrew))!
#> Using PKG_CFLAGS=-I/usr/local/include/postgresql@14
#> Using PKG_LIBS=-L/usr/local/lib/postgresql@14 -lpq
#> Using PKG_PLOGR=
#> ** libs
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumn.cpp -o DbColumn.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSource.cpp -o DbColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSourceFactory.cpp -o DbColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnStorage.cpp -o DbColumnStorage.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbDataFrame.cpp -o DbDataFrame.o
#> In file included from DbDataFrame.cpp:7:
#> vendor/boost/bind.hpp:36:1: warning: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior. [-W#pragma-messages]
#> BOOST_PRAGMA_MESSAGE(
#> ^
#> vendor/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
#> # define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
#>                                  ^
#> <scratch space>:39:2: note: expanded from here
#>  message("The practice of declaring the Bind placeholders (_1, _2, ...) " "in the global namespace is deprecated. Please use " "<boost/bind/bind.hpp> + using namespace boost::placeholders, " "or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.")
#>  ^
#> 1 warning generated.
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbResult.cpp -o DbResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSource.cpp -o PqColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSourceFactory.cpp -o PqColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqDataFrame.cpp -o PqDataFrame.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResult.cpp -o PqResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultImpl.cpp -o PqResultImpl.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultSource.cpp -o PqResultSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqUtils.cpp -o PqUtils.o
#> clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RPostgres-init.c -o RPostgres-init.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c connection.cpp -o connection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encode.cpp -o encode.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encrypt.cpp -o encrypt.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c logging.cpp -o logging.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c result.cpp -o result.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o RPostgres.so DbColumn.o DbColumnDataSource.o DbColumnDataSourceFactory.o DbColumnStorage.o DbConnection.o DbDataFrame.o DbResult.o PqColumnDataSource.o PqColumnDataSourceFactory.o PqDataFrame.o PqResult.o PqResultImpl.o PqResultSource.o PqUtils.o RPostgres-init.o RcppExports.o connection.o encode.o encrypt.o logging.o result.o -L/usr/local/lib/postgresql@14 -lpq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
#> ld: warning: dylib (/usr/local/lib/postgresql@14/libpq.dylib) was built for newer macOS version (12.0) than being linked (10.13)
#> installing to /Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-RPostgres/00new/RPostgres/libs
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** checking absolute paths in shared objects and dynamic libraries
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (RPostgres)
#> 
#> The downloaded source packages are in
#> 	‘/private/var/folders/np/n1zvy7cs7m9blq1w1_m_bqqw0000gn/T/RtmpTWKZBP/downloaded_packages’


# with libpq:
install.packages("RPostgres", type = "source")
#> trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.5.tar.gz'
#> Content type 'application/x-gzip' length 1473682 bytes (1.4 MB)
#> ==================================================
#> downloaded 1.4 MB
#> 
#> * installing *source* package ‘RPostgres’ ...
#> ** package ‘RPostgres’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> Homebrew 4.1.7
#> Homebrew/homebrew-core (git revision c1f15f343bb; last commit 2023-09-04)
#> Homebrew/homebrew-cask (git revision 7eae8590f2; last commit 2023-09-03)
#> Using PKG_CFLAGS=-I/usr/local/opt/libpq/include
#> Using PKG_LIBS=-L/usr/local/opt/libpq/lib -lpq
#> Using PKG_PLOGR=
#> ** libs
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumn.cpp -o DbColumn.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSource.cpp -o DbColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSourceFactory.cpp -o DbColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnStorage.cpp -o DbColumnStorage.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbDataFrame.cpp -o DbDataFrame.o
#> In file included from DbDataFrame.cpp:7:
#> vendor/boost/bind.hpp:36:1: warning: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior. [-W#pragma-messages]
#> BOOST_PRAGMA_MESSAGE(
#> ^
#> vendor/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
#> # define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
#>                                  ^
#> <scratch space>:39:2: note: expanded from here
#>  message("The practice of declaring the Bind placeholders (_1, _2, ...) " "in the global namespace is deprecated. Please use " "<boost/bind/bind.hpp> + using namespace boost::placeholders, " "or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.")
#>  ^
#> 1 warning generated.
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbResult.cpp -o DbResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSource.cpp -o PqColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSourceFactory.cpp -o PqColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqDataFrame.cpp -o PqDataFrame.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResult.cpp -o PqResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultImpl.cpp -o PqResultImpl.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultSource.cpp -o PqResultSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqUtils.cpp -o PqUtils.o
#> clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RPostgres-init.c -o RPostgres-init.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c connection.cpp -o connection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encode.cpp -o encode.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encrypt.cpp -o encrypt.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c logging.cpp -o logging.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c result.cpp -o result.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o RPostgres.so DbColumn.o DbColumnDataSource.o DbColumnDataSourceFactory.o DbColumnStorage.o DbConnection.o DbDataFrame.o DbResult.o PqColumnDataSource.o PqColumnDataSourceFactory.o PqDataFrame.o PqResult.o PqResultImpl.o PqResultSource.o PqUtils.o RPostgres-init.o RcppExports.o connection.o encode.o encrypt.o logging.o result.o -L/usr/local/opt/libpq/lib -lpq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
#> ld: warning: dylib (/usr/local/opt/libpq/lib/libpq.dylib) was built for newer macOS version (12.0) than being linked (10.13)
#> installing to /Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-RPostgres/00new/RPostgres/libs
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** checking absolute paths in shared objects and dynamic libraries
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (RPostgres)
#> 
#> The downloaded source packages are in
#> 	‘/private/var/folders/np/n1zvy7cs7m9blq1w1_m_bqqw0000gn/T/RtmpcjWwNf/downloaded_packages’

[krlmlr]

Is this still a problem with the most recent CRAN binary?

[krlmlr]

A duplicate of #404 perhaps?

[ilarischeinin]

Is this still a problem with the most recent CRAN binary?

It is.

Error: connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

I'm currently on macOS 14.4.1 and R 4.3.3, and it remains to be the case that the binary version (now 1.4.6) of RPostgres installed from CRAN does not work with our Redshift. When I compile RPostgres from source, it does work. (I have postgresql@14 version 14.11 installed via Homebrew, which is used when compiling.)

I'm on an Intel Mac, whereas the linked issue seemed to be specific to Apple Silicon, so I think this is something different.

While the problem is easy enough to fix for me by compiling form source, I recently had a colleague run into the same issue, and they were unable to compile from source on an Apple Silicon Mac. I didn't look into that in more detail yet, as I will be upgrading from Intel later this month. I will then see if I can get to a similar fix on Apple Silicon too, but of course it would be ideal if the CRAN binary worked out of the box.

One thing that has changed since originally reporting this is that it now affects both our Redshift instances (staging and production). So something (like an update, cert change, …) must have happened on the AWS side to the staging instance, as that used to work.

Something perhaps worth repeating is that the error happens before authentication, as the error message is the same regardless of whether I'm using valid username/password or not. (And is also different from the one I get with invalid credentials when using the compiled-from-source version of the package.)

[krlmlr]

I now understood Simon's comment.

Can you please try compiling with libpq from https://mac.r-project.org/bin/darwin20/x86_64/ (or whatever subdirectory is appropriate for your platform)? If we need a newer/different libpq, we could send a pull request to https://github.com/R-macos/recipes/blob/master/recipes/libpq .

[ilarischeinin]

OK, I:

1. removed any libpq and postgresql installations I had via brew
2. downloaded libpq-14.5-darwin.20-x86_64.tar.xz from the above link
3. placed the files inside the corresponding directories under /opt
4. compiled RPostgres from source with install.packages("RPostgres", type="source")

And as a result I am running into the issue described above. So I take it the hunch is correct there that libpq version used needs to be bumped from the current 14.5.

One thing I am confused by is that the compilation output mentions "autobrew bundle version 15.2":

> install.packages("RPostgres", type="source")
trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.6.tar.gz'
Content type 'application/x-gzip' length 1474867 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *source* package ‘RPostgres’ ...
** package ‘RPostgres’ successfully unpacked and MD5 sums checked
** using staged installation
Homebrew 4.2.18
Using autobrew bundle: libpq-15.2-universal.tar.xz

But the entire output doesn't make it clear to me if it's actually using the 14.5 I installed manually, or the 15.2 it might download as part of the compilation. There are lines that point to directories where I put the manual installation, e.g.

clude' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o

So my questions:

1. Could the source of the problem be the discrepancy (14.5 vs. 15.2), rather than simply an outdated version?
2. If I submit a PR to bump the version from the current 14.5, should I go for the 15.2 mentioned related to autobrew, or to 16.2, which looks to be the current both for brew and source? (And if 16.2, should something be done to also get the autobrew from 15.2 to 16.2?)

[krlmlr]

I believe the autobrew source is here: https://github.com/autobrew/scripts/blob/master/libpq . In some cases (High Sierra?), we still seem to be using 14.5 here. The autobrew mechanism can be disabled by setting the DISABLE_AUTOBREW environment variable.

The library used should be reported with DBI::dbGetInfo(RPostgres::Postgres()) . What are you seeing? I understand it doesn't work with 14.5, does it work for you with 15.2? I understand it works for you with 16.2, can you confirm?

@jeroen: Can/should we bump to 16.2 here? Would that work for the High Sierra case?

[jeroen]

High-sierra is no longer in use on CRAN or GHA or anywhere else so we don't need to worry about that.

I have updated the script to libpq 16.0 (for which big-sur binaries were already available): autobrew/scripts@ea29fea.

However, the problem is that the CRAN macbuilders now have another version of libpq 14.5 preinstalled on the server (via the recipes system) and therefore autobrew is not used. I think you can fix this by increasing to --atleast-version=15 in the configure script here:

RPostgres/configure

Lines 21 to 22 in 2b2081c

	if [ "`uname`" = Darwin ] && [ "`command -v pkg-config`" ]; then
	if pkg-config --atleast-version=12 libpq; then

[jeroen]

@ilarischeinin could you test the binary package from r-universe? If that one works, it is very likely that the next CRAN release will also work for you:

install.packages("RPostgres", repos = c("https://r-dbi.r-universe.dev", "https://cran.r-project.org"))