How to debug crashes

[alpha0010]

We have been using the manticoresearch/manticore:3.5.0 docker image, and experiencing an average of a crash every day (some days multiple, some days none at all). Decided to try the next version, manticoresearch/manticore:4.0.2, but the crashes still occur.

Log says to use:

addr2line -e searchd 0x5c01a0 0x47189f 0x52074890 0x51d22646 0x7c89d4 0x5d5032 0x5d57db 0x5d6875 
0x48b2ea 0x48b6a2 0x4ceb18 0x94fef7 0xd44b6c 0x48845c 0x489713 0x485f73 0x490a00 0x499be8 
0x46447e 0x462c8b 0x463872 0xd46cdc 0xd478ef > source.txt

but that is unusable, since official docker binaries have symbols stripped.

We do have a UDF. Has been working fine for the past 5-ish years (recompiled each time we updated sphinx/manticore). (Well actually, does not work on manticore 4.0.2. It runs without error, but produces wrong results. Probably not related to the crashes though.)

Unfortunately we do not know when the crashes started, nor how to reproduce them. Any suggestions how to debug?

[tomatolog]

could you try dev version of docker image docker pull manticoresearch/manticore:dev as 402-release have bug at RT index and could cause a crash

You could provide a searchd.log file with crash log and 10-20 lines prior it to investigate the crash.
You could also create your own image above https://github.com/manticoresoftware/docker/blob/master/Dockerfile but install symbols there or as a second layer

[alpha0010]

This manticore instance only uses plain indexes, so unlikely the problem. Will give it a try anyway.

Log of a crash: searchd.log

Are the symbols/unstripped binaries available for download from somewhere? Or would this be compile myself?

[alpha0010]

Here is a backtrace from docker manticoresearch/manticore:3.5.0. We cannot use 4.0 series in production due to #676.

searchd(sphBacktrace(int, bool)+0x9b)[0x556de6b29bab]
searchd(CrashLogger::HandleCrash(int)+0x1bb)[0x556de6999cbb]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f89e3ce3730]
/lib/x86_64-linux-gnu/libc.so.6(+0x15c181)[0x7f89e3720181]
searchd(sphParseSortClause(CSphQuery const*, char const*, ISphSchema const&, ESphSortFunc&, CSphMatchComparatorState&, bool, CSphString&)+0x57)[0x556de6b37cf7]
searchd(QueueCreator_c::SetupGroupSortingFunc(bool)+0x47)[0x556de6b48d67]
searchd(QueueCreator_c::SetGroupSorting()+0xb4)[0x556de6b48fd4]
searchd(sphCreateQueue(SphQueueSettings_t const&, CSphQuery const&, CSphString&, SphQueueRes_t&, sph::Vector_T<CSphString, sph::DefaultCopy_T<CSphString>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphString> >*)+0x2a)[0x556de6b490fa]
searchd(SearchHandler_c::CreateSingleSorters(CSphIndex const*, VecTraits_T<ISphMatchSorter*>&, VecTraits_T<CSphString>&, VecTraits_T<sph::Vector_T<CSphString, sph::DefaultCopy_T<CSphString>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphString> > >&, SphQueueRes_t&) const+0x7f)[0x556de69a066f]
searchd(SearchHandler_c::RunLocalSearches()+0x2b6)[0x556de69dc556]
searchd(SearchHandler_c::RunSubset(int, int)+0x1bd3)[0x556de69e0843]
searchd(SearchHandler_c::RunQueries()+0xb5)[0x556de69e0be5]
searchd(HandleCommandSearch(ISphOutputBuffer&, unsigned short, InputBuffer_c&)+0x231)[0x556de69e0eb1]
searchd(LoopClientSphinx(SearchdCommand_e, unsigned short, int, InputBuffer_c&, ISphOutputBuffer&, bool)+0x108)[0x556de69e1158]
searchd(ApiServe(SharedPtr_T<AsyncNetBuffer_c*, Deleter_T<AsyncNetBuffer_c*, (ETYPE)0>, ISphRefcountedMT>)+0x35c)[0x556de6a72f4c]
searchd(MultiServe(SharedPtr_T<SockWrapper_c*, Deleter_T<SockWrapper_c*, (ETYPE)0>, ISphRefcountedMT>, std::pair<int, unsigned short>)+0x10a)[0x556de6a71c5a]
searchd(+0x33a6d6)[0x556de6a726d6]
searchd(Threads::CoRoutine_c::CoRoutine_c(std::function<void ()>, unsigned long)::{lambda(boost::context::detail::transfer_t)#1}::_FUN(boost::context::detail::transfer_t)+0x1c)[0x556de6d77a9c]
searchd(make_fcontext+0x2f)[0x556de6d7d04f]

[alpha0010]

Trying to build from source (both 3.5.0 and 3.6.0), with no success. Any suggestions? Tried:

• registry.gitlab.com/manticoresearch/dev/bionic_cmake:320
  • docker: Error response from daemon: Head "https://registry.gitlab.com/v2/manticoresearch/dev/bionic_cmake/manifests/320": denied: access forbidden.
• Built the image myself based on https://github.com/manticoresoftware/manticoresearch/tree/master/dist/build_dockers/bionic
  • Still tried to download boost

[...]
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- WRITEB is not set, bundle will NOT be modified...
-- Boost 1.61.0 found.
-- Found Boost components:
   context
-- Try from previous run /manticore-src/build/boost-context-build
-- Boost 1.61.0 found.
-- Found Boost components:
   context
-- Try pre-built full boost from artifacts /cache/boost-linux-x86_64
-- Boost 1.61.0 found.
-- Found Boost components:
   context
-- Try pre-built boost from artifacts /cache/boost-context-linux-x86_64
-- Boost 1.61.0 found.
-- Found Boost components:
   context
-- Configuring done
-- Generating done
-- Build files have been written to: /manticore-src/build/boost-context-download
-- Download boost /manticore-src/build/boost-context-download
Scanning dependencies of target boost-context-download
[ 50%] Creating directories for 'boost-context'
[100%] Performing download step (download, verify and extract) for 'boost-context'
-- Downloading...
[...]

• Patched to use boostorg.jfrog.io/artifactory instead of dl.bintray.com.
  • Boost (re)compiled but failed to find itself

[...]
common.copy /cache/boost-linux-x86_64/include/boost/compatibility/cpp_c_headers/climits
common.copy /cache/boost-linux-x86_64/include/boost/compatibility/cpp_c_headers/cerrno
common.copy /cache/boost-linux-x86_64/include/boost/compatibility/cpp_c_headers/cassert
common.copy /cache/boost-linux-x86_64/include/boost/compatibility/cpp_c_headers/cctype
...updated 15918 targets...
[ 75%] Completed 'boost-context'
[100%] Built target boost-context
-- From fresh github sources /cache/boost-linux-x86_64
-- Boost 1.61.0 found.
-- Found Boost components:
   context
CMake Error at cmake/GetBoostContext.cmake:259 (message):
  Boost context is not found
Call Stack (most recent call first):
  CMakeLists.txt:124 (include)


-- Configuring incomplete, errors occurred!
See also "/manticore-src/build/CMakeFiles/CMakeOutput.log".

[tomatolog]

according to crash log the issue is at creating grouper for specific query. Could you check that query does not crash with your data at 40x version?

[alpha0010]

Crash was recorded on 4.0.2 (see searchd.log from #675 (reply in thread)). But as you can see from that log, the binary had no symbols, so not sure how to trace. Unfortunately do not know how to reproduce, since this is only noticed in production load.

(We did rebuild the index when switching versions, but they were built with the same config pulling from a mysql db.)

[tomatolog]

could you try current master head as 402 has its own issues these was fixed after

[sanikolaev]

We are working on some kind of FAQ, let me copy-paste the following here since it may be helpful in this case:

How do I install Manticore Search debug symbols?

Just install package *debuginfo* of exactly the same version you are running. For example if you've installed Manticore Search in Centos 8 from package https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-4.0.2_210921.af497f245-1.el8.x86_64.rpm the corresponding package with symbols is https://repo.manticoresearch.com/repository/manticoresearch/release/centos/8/x86_64/manticore-debuginfo-4.0.2_210921.af497f245-1.el8.x86_64.rpm

Note they have the same commit id af497f245 which corresponds to the commit this version was built from.

What are debug symbols at all and what can they be useful for?

Manticore Search and Manticore Columnar Library are written in C++, which means that what you get is a compiled compact binary file which executes in your OS optimal way. When you run a binary your system doesn't have full access to the names of variables, functions, methods, classes etc that are implemented. All that is provided separately in so called "debuginfo" packages or "symbol packages".
Debug symbols are useful for troubleshooting and other debugging purposes, since when you have symbols and your binary crashes there's a way to visualize the state it crashed at including function names. Manticore Search provides such backtrace in searchd log and also generates coredump if it was run with --coredump. Without symbols all you get is just internal offsets that is difficult/impossible to decode. So if you make a bug report about a crash in most cases Manticore team will need debug symbols to be able to help you.

How to enable saving coredumps on crash?

• make sure you run searchd with --coredump. To avoid hacking the scripts you can use this https://manual.manticoresearch.com/Starting_the_server/Linux#Custom-startup-flags-using-systemd , e.g.:

[root@srv lib]# systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS='--coredump'
[root@srv lib]# systemctl restart manticore
[root@srv lib]# ps aux|grep searchd
mantico+  1955  0.0  0.0  61964  1580 ?        S    11:02   0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump
mantico+  1956  0.6  0.0 392744  2664 ?        Sl   11:02   0:00 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf --coredump

• make sure that your OS allows you to save coredumps: /proc/sys/kernel/core_pattern should be non-empty - it is where it will save them. If you do:

echo "/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

it will instruct your kernel to save them to file like core.searchd.1773.centos-4gb-hel1-1.1636454937

• searchd should be started with ulimit -c unlimited, but if you start Manticore via systemctl it does it for yourself since it does:

[root@srv lib]# grep CORE /lib/systemd/system/manticore.service
LimitCORE=infinity

How to emulate a coredump?

Kill working searchd process with SIGSEGV (kernel sends the same signal when it detects segmentation fault):

[root@srv lib]# kill -SIGSEGV `cat /var/run/manticore/searchd.pid`

The searchd will be recreated immediately unless you disabled watchdog.
After that you should have a coredump created in the dir specified in /proc/sys/kernel/core_pattern :

[root@srv lib]# ls -la /cores/
total 1372
drwxrwxrwx.  2 root      root         4096 Nov  9 10:48 .
dr-xr-xr-x. 19 root      root         4096 Nov  9 10:47 ..
-rw-------.  1 manticore manticore 5599232 Nov  9 10:48 core.searchd.1773.srv.1636454937

How to check that debug symbols exist via gdb?

Install gdb and run it specifying path to the searchd binary and the coredump:

[root@srv lib]# gdb /usr/bin/searchd  /cores/core.searchd.1773.srv.1636454937
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/searchd...Reading symbols from /usr/lib/debug/usr/bin/searchd.debug...done.
done.
[New LWP 1773]
[New LWP 1774]
[New LWP 1778]
[New LWP 1775]
[New LWP 1776]
[New LWP 1777]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/searchd --config /etc/manticoresearch/manticore.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fafa40feb43 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-325.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 openssl-libs-1.0.2k-22.el7_9.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64
(gdb) bt
#0  0x00007fafa40feb43 in select () from /lib64/libc.so.6
#1  0x00000000004ff68f in sphSleepMsec (iMsec=<optimized out>) at ../src/sphinx.cpp:1555
#2  0x00000000004c6464 in TickHead () at ../src/searchd.cpp:18582
#3  0x00000000004d2f55 in ServiceMain (argc=<optimized out>, argv=<optimized out>) at ../src/searchd.cpp:19984
#4  0x00000000004d41cb in mainimpl (argc=0, argv=0x7ffcc63067b8) at ../src/searchd.cpp:20034
#5  main (argc=0, argv=0x7ffcc63067b8) at ../src/searchd.cpp:20040
(gdb) quit

You can see here Reading symbols from /usr/lib/debug/usr/bin/searchd.debug - it's good, it means you have symbols.

It also says debuginfo-install glibc-2.17-325.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 openssl-libs-1.0.2k-22.el7_9.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64. It's recommended to install these symbols too. Just execute the command gdb provides.

How to check the debug symbols exist another way?

1. Find build id in file /usr/bin/searchd output:

[root@srv lib]# file /usr/bin/searchd
/usr/bin/searchd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=2c582e9f564ea1fbeb0c68406c271ba27034a6d3, stripped

In this case it's 2c582e9f564ea1fbeb0c68406c271ba27034a6d3.

2. Find symbols in /usr/lib/debug/.build-id like this:

[root@srv ~]# ls -la /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3*
lrwxrwxrwx. 1 root root 23 Nov  9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3 -> ../../../../bin/searchd
lrwxrwxrwx. 1 root root 27 Nov  9 10:42 /usr/lib/debug/.build-id/2c/582e9f564ea1fbeb0c68406c271ba27034a6d3.debug -> ../../usr/bin/searchd.debug

What do I do when Manticore Search crashes?

When Manticore Search crashes you need to let Manticore team about that by making a bug report on github or if you use Manticore's professional services in your private helpdesk. Manticore team needs the following:

1. searchd log
2. coredump
3. query log

It will be great if you additionally do the following:

1. run gdb to inspect the coredump:

gdb /usr/bin/searchd </path/to/coredump>

2. Find crashed thread id in the coredump file name (make sure you have %p in /proc/sys/kernel/core_pattern), e.g. core.work_6.29050.server_name.1637586599 means thread_id=29050
3. In gdb run:

set pagination off
info threads
# find thread number by it's id (e.g. for `LWP 29050` it will be thread number 8 
thread apply all bt
thread <thread number>
bt full
info locals
quit

4. Provide the outputs

What do I do when Manticore Search hangs?

You need to run gdb manually and collect some info that may be useful to understand why it's hanging.

1. show threads option format=all run trough a VIP port
2. collect lsof output since hanging can be caused by too many connections or open file descriptors

lsof -p `cat /var/run/manticore/searchd.pid`

3. dump core

gcore `cat /var/run/manticore/searchd.pid`

(it will save the dump to the current dir)

4. Install and run gdb:

gdb /usr/bin/searchd `cat /var/run/manticore/searchd.pid

Note it will halt your running searchd, but if it's alredy hanging it shouldn't be a problem.
5. In gdb run:

set pagination off
info threads
thread apply all bt
quit

6. Collect all the outputs and files and provide them in a bug report.

For experts: the macros added in this commit can be helpful to debug.

[alpha0010]

So I apt install manticore-dbgsym in the 4.0.2 docker container, and was able to pull this trace from the above log file #675 (reply in thread).

# addr2line -e `which searchd` 0x5c01a0 0x47189f 0xdc17e890 0xdbe2c646 0x7c89d4 0x5d5032 0x5d57db 0x5d6875 0x48b2ea 0x48b6a2 0x4ceb18 0x94fef7 0xd44b6c 0x48845c 0x489713 0x485f73 0x490a00 0x499be8 0x46447e 0x462c8b 0x463872 0xd46cdc 0xd478ef
/builds/manticoresearch/dev/build/../src/sphinxutils.cpp:2821
/builds/manticoresearch/dev/build/../src/searchd.cpp:1130
??:0
??:0
/builds/manticoresearch/dev/build/../src/sortsetup.cpp:45
/builds/manticoresearch/dev/build/../src/sphinxsort.cpp:6534
/builds/manticoresearch/dev/build/../src/sphinxsort.cpp:6642
/builds/manticoresearch/dev/build/../src/sphinxsort.cpp:6833
/builds/manticoresearch/dev/build/../src/searchd.cpp:5525
??:0
/builds/manticoresearch/dev/build/../src/searchd.cpp:5825
??:?
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:275
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:275
/builds/manticoresearch/dev/build/../src/searchd.cpp:6850
/builds/manticoresearch/dev/build/../src/searchd.cpp:5462
??:0
??:0
/builds/manticoresearch/dev/build/../src/netreceive_api.cpp:186
/builds/manticoresearch/dev/build/../src/sphinxstd.h:4023
/builds/manticoresearch/dev/build/../src/sphinxstd.h:4023
/builds/manticoresearch/dev/build/../src/coroutine.cpp:86
??:?

Trying current master head/getting a core dump will have to wait until next year. (Something about "don't mess with production when the site is at peak usage"...)