compile Mbed TLS from package source instead of using incompatible system version?

[tdhock]

Hi! I'm trying to install from source on Linux. I expected this should be possible, but I get an error message "undefined symbol: mbedtls_ctr_drbg_random" which seems to indicate there is a problem with a mbedtls function.
Output below indicates that the build script found mbedtls 2.28.1. I see that version is not compatible with the version stated in SystemRequirements: 'libnng' >= 1.9 and 'libmbedtls' >= 2.5, or 'cmake' and 'xz' to compile NNG and/or Mbed TLS included in package sources.

This indicates that Mbed TLS is included with package sources, but it is trying to use an incompatible version that is installed on my system.

I would have expected that if an incompatible version is found, the build script should decide to compile Mbed TLS from package source, instead of trying to use the incompatible system version. Could that change be made, please?

Full installation output below.

[thocking@beluga2 downloaded_packages]$ tar xf nanonext_1.6.0.tar.gz 
[thocking@beluga2 downloaded_packages]$ R CMD INSTALL nanonext
* installing to library '/home/thocking/R/x86_64-pc-linux-gnu-library/4.4'
* installing *source* package 'nanonext' ...
** package 'nanonext' successfully unpacked and MD5 sums checked
** using staged installation
Existing 'libnng' >= 1.9.0 not found
Detecting 'cmake'...
/cvmfs/soft.computecanada.ca/custom/bin/cmake
Detecting 'xz'...
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/xz
Compiling 'libnng' from source ...
-- The C compiler identification is GNU 12.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/\
x86_64-pc-linux-gnu/gcc-bin/12/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring for NNG version 1.10.2-pre
-- Performing Test NNG_HIDDEN_VISIBILITY
-- Performing Test NNG_HIDDEN_VISIBILITY - Success
-- Looking for strlcpy
-- Looking for strlcpy - not found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for strcasecmp
-- Looking for strcasecmp - found
-- Looking for strncasecmp
-- Looking for strncasecmp - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for lockf
-- Looking for lockf - found
-- Looking for flock
-- Looking for flock - found
-- Looking for getrandom
-- Looking for getrandom - found
-- Looking for arc4random_buf
-- Looking for arc4random_buf - found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for sem_wait in pthread
-- Looking for sem_wait in pthread - found
-- Looking for pthread_atfork in pthread
-- Looking for pthread_atfork in pthread - found
-- Looking for pthread_set_name_np in pthread
-- Looking for pthread_set_name_np in pthread - not found
-- Looking for pthread_setname_np in pthread
-- Looking for pthread_setname_np in pthread - found
-- Looking for gethostbyname in nsl
-- Looking for gethostbyname in nsl - found
-- Looking for socket in socket
-- Looking for socket in socket - not found
-- Looking for __atomic_load_1 in atomic
-- Looking for __atomic_load_1 in atomic - found
-- Looking for AF_UNIX - found
-- Looking for backtrace_symbols_fd
-- Looking for backtrace_symbols_fd - found
-- Performing Test NNG_HAVE_MSG_CONTROL
-- Performing Test NNG_HAVE_MSG_CONTROL - Success
-- Looking for eventfd
-- Looking for eventfd - found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Looking for port_create
-- Looking for port_create - not found
-- Looking for epoll_create
-- Looking for epoll_create - found
-- Looking for epoll_create1
-- Looking for epoll_create1 - found
-- Looking for getpeereid
-- Looking for getpeereid - not found
-- Looking for SO_PEERCRED
-- Looking for SO_PEERCRED - found
-- Performing Test NNG_HAVE_SOCKPEERCRED
-- Performing Test NNG_HAVE_SOCKPEERCRED - Failed
-- Looking for LOCAL_PEERCRED
-- Looking for LOCAL_PEERCRED - not found
-- Looking for LOCAL_PEERPID
-- Looking for LOCAL_PEERPID - not found
-- Looking for getpeerucred
-- Looking for getpeerucred - not found
-- Looking for atomic_flag_test_and_set
-- Looking for atomic_flag_test_and_set - found
-- Looking for socketpair
-- Looking for socketpair - found
-- Found mbedTLS: /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libmbedtl\
s.so (found version "2.28.1")
-- Configuring done (5.9s)
-- Generating done (0.1s)
-- Build files have been written to: /tmp/RtmpKScC18/downloaded_packages/nanonext/nng-11\
0-2a
[  0%] Building C object CMakeFiles/nng.dir/src/nng.c.o
[  2%] Building C object CMakeFiles/nng.dir/src/core/aio.c.o
[  2%] Building C object CMakeFiles/nng.dir/src/core/device.c.o
[  4%] Building C object CMakeFiles/nng.dir/src/core/dialer.c.o
...
[ 98%] Building C object CMakeFiles/nng.dir/src/supplemental/websocket/websocket.c.o
[100%] Linking C static library libnng.b
[100%] Built target nng
Install the project...
-- Install configuration: "Release"
** libs
using C compiler: 'gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526'
gcc -I"/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcccore/r\
/4.4.0/lib64/R/include" -DNDEBUG   -I/cvmfs/soft.computecanada.ca/easybuild/software/202\
3/x86-64-v3/Core/java/17.0.6/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2\
023/x86-64-v3/Core/flexiblascore/3.3.1/include -I/cvmfs/soft.computecanada.ca/easybuild/\
software/2023/x86-64-v3/Core/flexiblascore/3.3.1/include/flexiblas   -I../install/includ\
e  -fvisibility=hidden -fpic  -O2 -ftree-vectorize -march=x86-64-v4 -fno-math-errno  -c \
aio.c -o aio.o
...
gcc -I"/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gcccore/r\
/4.4.0/lib64/R/include" -DNDEBUG   -I/cvmfs/soft.computecanada.ca/easybuild/software/202\
3/x86-64-v3/Core/java/17.0.6/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2\
023/x86-64-v3/Core/flexiblascore/3.3.1/include -I/cvmfs/soft.computecanada.ca/easybuild/\
software/2023/x86-64-v3/Core/flexiblascore/3.3.1/include/flexiblas   -I../install/includ\
e  -fvisibility=hidden -fpic  -O2 -ftree-vectorize -march=x86-64-v4 -fno-math-errno  -c \
utils.c -o utils.o
gcc -shared -L/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v4/Compiler/gc\
ccore/r/4.4.0/lib64/R/lib -o nanonext.so aio.o comms.o core.o init.o ncurl.o net.o proto\
.o sync.o thread.o tls.o utils.o ../install/lib/libnng.b -L/cvmfs/soft.computecanada.ca/\
easybuild/software/2023/x86-64-v4/Compiler/gcccore/r/4.4.0/lib64/R/lib -lR
installing to /home/thocking/R/x86_64-pc-linux-gnu-library/4.4/00LOCK-nanonext/00new/nan\
onext/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for 'nanonext' in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/thocking/R/x86_64-pc-linux-gnu-library/4.4/00LOCK-nanonext/00new/nanonext/libs/nanonext.so':
  /home/thocking/R/x86_64-pc-linux-gnu-library/4.4/00LOCK-nanonext/00new/nanonext/libs/nanonext.so: undefined symbol: mbedtls_ctr_drbg_random
Error: loading failed
Execution halted
ERROR: loading failed
* removing '/home/thocking/R/x86_64-pc-linux-gnu-library/4.4/nanonext'
[thocking@beluga2 downloaded_packages]$ 

[tdhock]

@shikokuchuo can you help please?

[tdhock]

Actually on second thought the system version, 2.28.1, seems to be more recent than the version listed in SystemRequirements (>= 2.5)... is that right?
If so then what is the issue?

[shikokuchuo]

Very strange - the location of mbedtls is in a non-standard location that the configure script should not find, and hence it should just build it from source.

Conversely if it's found you should see something like the following as the first message (with the path where it's found):

** using staged installation
Found 'libmbedtls' -I/usr/local/include

Whereas your output jumps straight to:

** using staged installation
Existing 'libnng' >= 1.9.0 not found

To just get it to work:
Sys.setenv(NANONEXT_LIBS=1) will skip detection of system libs and force building from source.

[shikokuchuo]

Hmm seems this may have been introduced prior to the last 1.5.2 release - the linker flags are no longer set when the detection in the configure script fails.

I've pushed a patch to the 'configure' branch. This should now work assuming both the header and library paths are being picked up by the compiler.

@tdhock can you test if that works for you, e.g. by:

pak::pak("r-lib/nanonext@configure")

Thanks!

[tdhock]

Thanks for your suggestion about Sys.setenv(NANONEXT_LIBS=1) that worked for me.

[thocking@beluga2 downloaded_packages]$ NANONEXT_LIBS=1 R CMD INSTALL nanonext_1.6.0.tar.gz 
* installing to library '/home/thocking/R/x86_64-pc-linux-gnu-library/4.4'
* installing *source* package 'nanonext' ...
** package 'nanonext' successfully unpacked and MD5 sums checked
** using staged installation
NANONEXT_LIBS is set... skipping detection
Existing 'libmbedtls' >= 2.5.0 not found
...
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (nanonext)
[thocking@beluga2 downloaded_packages]$ 

[tdhock]

Thanks for the quick update. I tried it and it seems to work!

> remotes::install_github("r-lib/nanonext@configure")
 Downloading GitHub repo r-lib/nanonext@configure
 -- R CMD build ------------------------------------------------------------------------
 v  checking for file '/tmp/RtmpdayRI2/remotes358d7d20a5b4ef/r-lib-nanonext-f7c72b6/DESC\
 RIPTION' (451ms)                                                                        
 -  preparing 'nanonext':                                                                
 v  checking DESCRIPTION meta-information                                                
 -  cleaning src                                                                         
 -  running 'cleanup'                                                                    
 -  checking for LF line-endings in source and make files and shell scripts              
 -  checking for empty or unneeded directories                                           
 -  building 'nanonext_1.6.0.9000.tar.gz'                                                
    Warning: file 'nanonext/cleanup' did not have execute permissions: corrected         
    Warning: file 'nanonext/configure' did not have execute permissions: corrected       
    Warning: invalid uid value replaced by that for user 'nobody'                        
    Warning: invalid gid value replaced by that for user 'nobody'                        
                                                                                         
 Installing package into '/home/thocking/R/x86_64-pc-linux-gnu-library/4.4'              
 (as 'lib' is unspecified)                                                               
 * installing *source* package 'nanonext' ...                                            
 ** using staged installation                                                            
 Found 'libmbedtls'                                                                      
 Existing 'libnng' >= 1.9.0 not found                                                    
 Detecting 'cmake'...                                                                    
...
 installing to /home/thocking/R/x86_64-pc-linux-gnu-library/4.4/00LOCK-nanonext/00new/na\
 nonext/libs                                                                             
 ** R                                                                                    
 ** inst                                                                                 
 ** byte-compile and prepare package for lazy loading                                    
 ** help                                                                                 
 *** installing help indices                                                             
 *** copying figures                                                                     
 ** building package indices                                                             
 ** installing vignettes                                                                 
 ** testing if installed package can be loaded from temporary location                   
 ** checking absolute paths in shared objects and dynamic libraries                      
 ** testing if installed package can be loaded from final location                       
 ** testing if installed package keeps a record of temporary installation path           
 * DONE (nanonext)                                                                       
 >