Fix popen leak and handle perf script crash

[mlim19]

The patch is revised to take care of cleaning up the popen object being still referenced after the associated subprocess gets terminated.

Based on code review, 3 issues are identified:

1. In run_process function, If the assertion does NOT happen, the subprocess is terminated. At this time, I think we should remove the popen object in the global list so that the pipes and file descriptors are cleaned by GC.

• a. With python 3.2 or later, subprocess.Popen supports the context manager protocol for with statement, which means exit method is implemented and executed when t the with block ends. Popen.exit calls proc.wait() by default.
• b. At the end of the with block, it looks we ensures the process is terminated by checking the retcode from poll().

  gprofiler/gprofiler/utils/__init__.py

  Line 266 in 63ef435

  assert retcode is not None # only None if child has not terminated

2. Some profilers (php, pyperf, perf) call the start_process function directly. So, the above solution for run_process is not helpful. So, we handle the clean up separately here
3. When "perf script" is crashed, we need to add the handling of exception which should fixes CPU profiling on GPU machines #992

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots

Checklist:

• I have read the CONTRIBUTING document.
• I have updated the relevant documentation.
• I have added tests for new logic.

[dkorlovs]

thank you!

[mlim19]

Added a new patch to take care of the cases not handled. After applying the patchset collectively, the overall memory utilization remain same over several hours.

[prashantbytesyntax]

The fix looks good, there is still memory growth which i think needs handling for runtime profilers and perf output parsing optimization but could be done in future PRs

[prashantbytesyntax]

we probably need clean up on run time profilers too