How to analyze a Linux "core" file (i.e. crash dump) #199
Description
How to get a core dump
ulimit -a (lists all limits)
ulimit -c (core dump size)
ulimit -c unlimited (enable core dumps)
ulimit is a shell builtin, and thus only affects the current shell and processes started by that shell. To set limits permanently or for all processes, edit the file /etc/security/limits.conf and reboot. The examples in the limits.conf manpage are fairly good. You just need to add something like:
# gedit /etc/security/limits.conf
. . .
myusrid - core unlimited
Where are "core" files written?
If you run Hercules as root, they appear to be placed in the current directory as a hidden file (because the owner is root):
$ ls -al ~/hercules/hercules-0/core* -rw-------. 1 root root 104226816 Apr 6 01:24 /home/fish/hercules/hercules-0/core.56278
If you run Hercules as a regular user I'm guessing they'll be placed in the current directory as a regular file.
Obtaining a backtrace from a "core" file
You can open a core file with gdb like this:
$ gdb /path/to/executable /path/to/core/file
which should then display exactly where the crash occurred and which thread it was.
For the Hercules case, since it uses libtool, the path to the executable is typically something like ".libs/lt-hercules".
To see what each thread was doing when the crash occurred (including the one that crashed), use the gdb command:
(gdb) thread apply all bt
which should then display a full backtrace for each thread, identifying the source file and line number of each of the thread's function calls.
If you want to save the output of your gdb session to a log file (recommended!), issue the following gdb command as your very first gdb command (e.g. before you issue your gdb backtrace command):
(gdb) set logging on
For more information regarding gdb's logging capabilities, please see:
Obtaining a backtrace by running Hercules directly under gdb itself
If the crash is reproducible, you can start Hercules directly from the gdb debugger as follows:
$ gdb .libs/lt-hercules (program to be debugged) (gdb) run -f myhercconfig -o myherclogfile (command line arguments) (gdb) backtrace
To avoid signal noise (e.g. if gdb breaks on a SIGUSR2 event):
Thread 10 "LCS_PortThread" received signal SIGUSR2, User defined signal 2.
[Switching to Thread 0x7fffd77ca700 (LWP 28243)]
0x00007ffff5be3f2c in close () from /lib64/libpthread.so.0
It appears you can do either:
- Press
cto continue whenever the SIGUSR2 break occurs. - Enter the gdb command
handle SIGUSR2 noprint nostopwhen gdb is first started. - Both.