Add results from biolatency, {xfs,ext4}slower, and cachestat (bcc tools)

[rustyrazorblade]

if bcc tools are installed and available it would be great to capture their results. I've found them to be more more useful when debugging something that's IO bound than tools like iostat which summarizes data in a way that hides outliers.

biolatency can provide a histogram of block level latency response times:

./biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs           : count     distribution
       0 -> 1        : 0        |                                      |
       2 -> 3        : 0        |                                      |
       4 -> 7        : 0        |                                      |
       8 -> 15       : 0        |                                      |
      16 -> 31       : 0        |                                      |
      32 -> 63       : 0        |                                      |
      64 -> 127      : 1        |                                      |
     128 -> 255      : 12       |********                              |
     256 -> 511      : 15       |**********                            |
     512 -> 1023     : 43       |*******************************       |
    1024 -> 2047     : 52       |**************************************|
    2048 -> 4095     : 47       |**********************************    |
    4096 -> 8191     : 52       |**************************************|
    8192 -> 16383    : 36       |**************************            |
   16384 -> 32767    : 15       |**********                            |
   32768 -> 65535    : 2        |*                                     |
   65536 -> 131071   : 2        |*                                     |

The *slower utilities give similar information but for a filesystem:

# ./xfsslower
Tracing XFS operations slower than 10 ms
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
06:23:06 randread.pl    32497  R 8192    24938024   17.93 data1
06:23:06 randread.pl    32521  R 8192    13431528   18.27 data1
06:23:08 randread.pl    32497  R 8192    5070904    16.37 data1
06:23:08 randread.pl    32521  R 8192    12693016   16.06 data1
06:23:18 randread.pl    32521  R 8192    27049136   21.68 data1
06:23:18 randread.pl    32497  R 8192    257864     21.74 data1
06:23:20 randread.pl    32497  R 8192    17797208   13.37 data1
06:23:20 randread.pl    32497  R 8192    6088224    19.74 data1

Lastly, cachestat is fantastic for determining if a config change giving more memory to a program resulted in a change to cache hit rate:

# cachestat
    HITS   MISSES  DIRTIES HITRATIO   BUFFERS_MB  CACHED_MB
    1132        0        4  100.00%          277       4367
     161        0       36  100.00%          277       4372
      16        0       28  100.00%          277       4372
   17154    13750       15   55.51%          277       4422
      19        0        1  100.00%          277       4422
      83        0       83  100.00%          277       4421
      16        0        1  100.00%          277       4423
^C       0      -19      360    0.00%          277       4423
Detaching...

There's quite a few helpful utilities in here that touch every aspect of a running program, but these are the ones I use the most when digging into performance issues with databases.

There's two downsides to these tools:

1. Running them generally requires root
2. They have different install locations and sometimes different names depending on the distro. If you decide to add this functionality I'd be happy to help identify the names and install locations. For now I suppose it's probably best to support the common case (RH 8.1 and up, Ubuntu) where it's installed in /usr/share/bcc/tools.

Additional reading: https://www.redhat.com/en/blog/bcc-tools-brings-dynamic-kernel-tracing-red-hat-enterprise-linux-81

[ycrash]

Hello Jon Haddad, Thank you for reaching out to us and providing valuable feedback on our application. We sincerely appreciate the time and effort you took to point out these additional tools that could enhance our toolset. We are grateful for your suggestions and will evaluate them carefully to determine if they can be incorporated into our repository in the upcoming releases. Regarding your suggestions for capturing the results of bcc tools, we understand their usefulness, especially when debugging IO-bound issues. The tools you mentioned, such as biolatency, *slower utilities, and cachestat, seem to offer valuable insights into block-level latency response times, filesystem performance, and cache hit rates. We acknowledge their significance in performance analysis, particularly for databases. We genuinely value your input and welcome further suggestions and feedback to continuously improve our toolset. If you come across any other tools or have additional ideas that could contribute to better performance analysis, please do not hesitate to share them with us. Your expertise and insights are highly appreciated. Thank you once again for your contribution, and we look forward to further collaboration. *Malini Valli | +1 (415) 948 5431 * *Dramatically improve application performance. Drastically reduce troubleshooting time.* *y*Crash | GCeasy | fastThread *| **HeapHero*

…

On Fri, Jul 7, 2023 at 1:48 AM Jon Haddad ***@***.***> wrote: if bcc tools are installed and available it would be great to capture their results. I've found them to be more more useful when debugging something that's IO bound than tools like iostat which summarizes data in a way that hides outliers. biolatency can provide a histogram of block level latency response times: ./biolatency Tracing block device I/O... Hit Ctrl-C to end. ^C usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 1 | | 128 -> 255 : 12 |******** | 256 -> 511 : 15 |********** | 512 -> 1023 : 43 |******************************* | 1024 -> 2047 : 52 |**************************************| 2048 -> 4095 : 47 |********************************** | 4096 -> 8191 : 52 |**************************************| 8192 -> 16383 : 36 |************************** | 16384 -> 32767 : 15 |********** | 32768 -> 65535 : 2 |* | 65536 -> 131071 : 2 |* | The *slower utilities give similar information but for a filesystem: # ./xfsslower Tracing XFS operations slower than 10 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 06:23:06 randread.pl 32497 R 8192 24938024 17.93 data1 06:23:06 randread.pl 32521 R 8192 13431528 18.27 data1 06:23:08 randread.pl 32497 R 8192 5070904 16.37 data1 06:23:08 randread.pl 32521 R 8192 12693016 16.06 data1 06:23:18 randread.pl 32521 R 8192 27049136 21.68 data1 06:23:18 randread.pl 32497 R 8192 257864 21.74 data1 06:23:20 randread.pl 32497 R 8192 17797208 13.37 data1 06:23:20 randread.pl 32497 R 8192 6088224 19.74 data1 # cachestat HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB 1132 0 4 100.00% 277 4367 161 0 36 100.00% 277 4372 16 0 28 100.00% 277 4372 17154 13750 15 55.51% 277 4422 19 0 1 100.00% 277 4422 83 0 83 100.00% 277 4421 16 0 1 100.00% 277 4423 ^C 0 -19 360 0.00% 277 4423 Detaching... There's quite a few helpful utilities in here that touch every aspect of a running program, but these are the ones I use the most when digging into performance issues with databases. There's two downsides to these tools: 1. Running them generally requires root 2. They have different install locations and sometimes different names depending on the distro. If you decide to add this functionality I'd be happy to help identify the names and install locations. For now I suppose it's probably best to support the common case (RH 8.1 and up, Ubuntu) where it's installed in /usr/share/bcc/tools. Additional reading: https://www.redhat.com/en/blog/bcc-tools-brings-dynamic-kernel-tracing-red-hat-enterprise-linux-81 — Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUDYTYRUHGGIELOZ6LLOY7DXO4MQLANCNFSM6AAAAAA2A5IN2M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>