Failed to run heap on Fedora:31

[fridex]

I've tried top setup gdb-heap on my Fedora:31 machine. It feels to me gdb-heap initializes correctly and I'm able to attach to an existing process, gdb-heap commands are registered and most of them work. The issue is when I'm trying to obtain memory information using heap command:

$ gdb -p 127637 -x gdb-heap-commands
GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 127637
[New LWP 127648]
[New LWP 127649]
[New LWP 127650]
[New LWP 127651]
[New LWP 127652]
[New LWP 127653]
[New LWP 127654]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/numpy/core/../../numpy.libs/libgfortran-ed201abd.so.3.0.0
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libgssapi_krb5-174f8956.so.2.2
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libz-a147dcb0.so.1.2.3
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libkrb5-fb0d2caa.so.3.3
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libk5crypto-622ef25b.so.3.1
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libcom_err-beb60336.so.2.1
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libkrb5support-d7ce89d4.so.0.1
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libkeyutils-1-ff31573b.2.so
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libselinux-cf8f9094.so.1
Missing separate debuginfo for /home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib64/python3.6/site-packages/psycopg2/../psycopg2_binary.libs/./libsepol-b4f5b513.so.1
0x00007f65a169da3f in __GI___poll (fds=0x7ffffe9223f0, nfds=1, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
29	  return SYSCALL_CANCEL (poll, fds, nfds, timeout);
(gdb) heap sizes
Chunk size  Num chunks  Allocated size
----------  ----------  --------------
       944        6093       5,751,792
     1,120        3013       3,374,560
       592        4443       2,630,256
     2,224         682       1,516,768
     4,640         316       1,466,240
       544        1591         865,504
    36,912          14         516,768
     9,264          49         453,936
     4,288          79         338,752
    18,464          18         332,352
     4,400          75         330,000
       576         503         289,728
     4,384          64         280,576
       976         234         228,384
     4,272          52         222,144
    73,760           3         221,280
     4,304          50         215,200
     2,064         100         206,400
       624         327         204,048
     8,208          24         196,992
    32,832           6         196,992
       560         305         170,800
       720         230         165,600
       896         169         151,424
       608         249         151,392
       672         223         149,856
       528         281         148,368
     4,416          33         145,728
       640         213         136,320
     1,344          97         130,368
       688         180         123,840
     1,744          70         122,080
     1,024         118         120,832
     1,280          92         117,760
     1,728          67         115,776
    16,400           7         114,800
       656         174         114,144
        32        3544         113,408
       704         156         109,824
     2,256          48         108,288
     1,104          97         107,088
     1,392          76         105,792
       784         130         101,920
     1,056          95         100,320
       800         125         100,000
     4,320          23          99,360
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) heap
Blocks retrieved 10000
Blocks retrieved 20000
Blocks retrieved 30000
Python Exception <class 'gdb.error'> There is no member named str.: 
Error occurred in Python: There is no member named str.
(gdb) 

Installed packages:

gdb-heap-0.5-36.20191013gitf3dcc53.fc31.x86_64
glib2-2.62.6-1.fc31.x86_64
glib2-devel-2.62.6-1.fc31.x86_64

Running Python3.6.

Thanks for any pointers.

[fridex]

After digging further, it looks like the gdb python interface has changed, more precisely native _gdb and its Value has changed and it no longer provides str:

(gdb) set python print-stack full
(gdb) heap
Blocks retrieved 10000
Blocks retrieved 20000
Blocks retrieved 30000
Traceback (most recent call last):
  File "/usr/share/gdb-heap/heap/commands.py", line 34, in g
    return f(self, args, from_tty)
  File "/usr/share/gdb-heap/heap/commands.py", line 56, in invoke
    usage_list = list(lazily_get_usage_list())
  File "/usr/share/gdb-heap/heap/__init__.py", line 503, in lazily_get_usage_list
    categorize_usage_list(usage_list)
  File "/usr/share/gdb-heap/heap/__init__.py", line 528, in categorize_usage_list
    if u.obj.categorize_refs(usage_set):
  File "/usr/share/gdb-heap/heap/cpython.py", line 280, in categorize_refs
    m_str = int(self.field('str'))
  File "/usr/share/gdb-heap/heap/__init__.py", line 117, in field
    return self._gdbval[attr]
gdb.error: There is no member named str.
Error occurred in Python: There is no member named str.

I patched the corresponding part:

gdb-heap/heap/cpython.py

Lines 279 to 284 in b0a6efc

	def categorize_refs(self, usage_set, level=0, detail=None):
	m_str = int(self.field('str'))
	usage_set.set_addr_category(m_str,
	Category('cpython', 'PyUnicodeObject buffer', detail),
	level)
	return True

And provided an alternative implementation:

    def categorize_refs(self, usage_set, level=0, detail=None):
        usage_set.set_addr_category(self.as_address(),  # << directly obtain as an address
                                    Category('cpython', 'PyUnicodeObject buffer', detail),
                                    level)
        return True

That lead to another issue spotted:

(gdb) heap
Blocks retrieved 10000
Blocks retrieved 20000
Blocks retrieved 30000
Traceback (most recent call last):
  File "/usr/share/gdb-heap/heap/commands.py", line 34, in g
    return f(self, args, from_tty)
  File "/usr/share/gdb-heap/heap/commands.py", line 56, in invoke
    usage_list = list(lazily_get_usage_list())
  File "/usr/share/gdb-heap/heap/__init__.py", line 504, in lazily_get_usage_list
    categorize_usage_list(usage_list)
  File "/usr/share/gdb-heap/heap/__init__.py", line 529, in categorize_usage_list
    if u.obj.categorize_refs(usage_set):
  File "/usr/share/gdb-heap/heap/cpython.py", line 286, in categorize_refs
    level)
  File "/usr/share/gdb-heap/heap/__init__.py", line 365, in set_addr_category
    if level <= u.level:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
Error occurred in Python: '<=' not supported between instances of 'int' and 'NoneType'
(gdb) 

After changing the condition:

gdb-heap/heap/__init__.py

Lines 364 to 365 in b0a6efc

	if level <= u.level:
	if debug:

to:

            if u.level is not None and level <= u.level:

I obtained the following error:

(gdb) heap
Blocks retrieved 10000
Blocks retrieved 20000
Blocks retrieved 30000
Blocks analyzed 10000
Traceback (most recent call last):
  File "/usr/share/gdb-heap/heap/commands.py", line 34, in g
    return f(self, args, from_tty)
  File "/usr/share/gdb-heap/heap/commands.py", line 56, in invoke
    usage_list = list(lazily_get_usage_list())
  File "/usr/share/gdb-heap/heap/__init__.py", line 503, in lazily_get_usage_list
    categorize_usage_list(usage_list)
  File "/usr/share/gdb-heap/heap/__init__.py", line 528, in categorize_usage_list
    if u.obj.categorize_refs(usage_set):
  File "/usr/share/gdb-heap/heap/cpython.py", line 297, in categorize_refs
    ma_table = int(self.field('ma_table'))
  File "/usr/share/gdb-heap/heap/__init__.py", line 117, in field
    return self._gdbval[attr]
gdb.error: There is no member named ma_table.
Error occurred in Python: There is no member named ma_table.

To my understanding, gdb-heap is not compatible with dict implementation provided by cpython interpreter version I'm running (Python 3.6).

Any pointers or comments welcomed.

[edmcman]

Yes, the dict implementation has changed. I patched things enough at https://github.com/edmcman/gdb-heap to avoid crashing, though I didn't fully iterate through the new data structures.