perf-tools: assess performance overhead

[gireeshpunathil]

the performance overhead of this tool should be measured and documented.

• under default configurations
• how does it vary with sampling rate
• how does it vary with various collection of data points

[gireeshpunathil]

right now the function GetOSThreadID (COM_IBM_GET_OS_THREAD_ID) is discovered dynamically, on a per monitor event call. This can be improved, by taking this out of the callbacks, performed in init, as a one time activity?

[mpirvu]

This can be improved, by taking this out of the callbacks, performed in init, as a one time activity?
Absolutely, I had this is mind. All JVMTI extensions should be discovered and cached.

[mpirvu]

Liberty start-up time is regressed 16% by adding the agent to the command line.
Linux perf shows this overhead:

  37.41%        183970  libagent.so
  30.98%        152344  libj9jit29.so
  13.81%         67905  libj9vm29.so
   4.62%         22709  libj9gc29.so
   3.41%         16773  [JIT] tid 19391
   3.14%         15466  [kernel.kallsyms]
   1.92%          9446  libj9thr29.so
   1.28%          6319  libc-2.27.so
   1.21%          5957  ld-2.27.so
   0.74%          3660  libpthread-2.27.so
   0.54%          2641  libz.so.1.2.11

[mpirvu]

Ticks in the agent:

   7.14%         35121  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::begin
   6.14%         30193  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::empty
   5.66%         27841  main     [.] __gnu_cxx::operator==<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >
   5.34%         26244  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::end
   5.03%         24716  main     [.] __gnu_cxx::__normal_iterator<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >::__normal_iterator
   3.18%         15630  main     [.] __gnu_cxx::__normal_iterator<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >::base
   2.83%         13933  main     [.] Server::handleServer
   0.84%          4132  main     [.] __gnu_cxx::__normal_iterator<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >::base@plt
   0.45%          2192  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::begin@plt
   0.44%          2160  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::end@plt
   0.19%           953  main     [.] __gnu_cxx::__normal_iterator<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >::__normal_iterator@plt
   0.17%           841  main     [.] std::vector<delayed_command_t, std::allocator<delayed_command_t> >::empty@plt
   0.00%            10  main     [.] __gnu_cxx::operator==<delayed_command_t const*, std::vector<delayed_command_t, std::allocator<delayed_command_t> > >@plt
   0.00%             1  java     [.] std::shared_ptr<std::__detail::_NFA<std::__cxx11::regex_traits<char> > const>::shared_ptr
   0.00%             1  main     [.] nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned lon
   0.00%             1  main     [.] std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<c
   0.00%             1  main     [.] std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::cha