Profiling #260
Profiling #260
Conversation
This commit adds the needed bits for the main Python process to create a peer cgroup (linux only) such that when profiling is enabled, the PRM containers are run under this cgroup with the `memory.peak` and `cpu.stat` controllers enabled. Unfortunately we can't just point Python at some PID, because the PRM containers launch various processes without reporting the PIDs back to the originating process. This prevents us from regular inline monitoring.
|
|
||
| # Write the contents of the file | ||
| write_header = not os.path.exists(profile_path) or os.path.getsize(profile_path) == 0 | ||
| with open(profile_path, "a", newline="") as out_f: |
There was a problem hiding this comment.
A test on apptainer for testing for the existence of profile_path would be good.
| @@ -240,14 +242,98 @@ def run_container_docker(container: str, command: List[str], volumes: List[Tuple | |||
| return out | |||
|
|
|||
|
|
|||
| def run_container_singularity(container: str, command: List[str], volumes: List[Tuple[PurePath, PurePath]], working_dir: str, environment: str = 'SPRAS=True'): | |||
| def create_cgroup() -> str: | |||
There was a problem hiding this comment.
[I'm generally made uncomfortable with including such a specific function in this general containers file, and I would like to see profiling in general moved to a separate file (especially if we decided to add profiling for other platforms/container frameworks). Understandably, though, that is a little much.]
| path_elements = container.split("/") | ||
|
|
||
| # Get the last element, which will indicate the base container name |
There was a problem hiding this comment.
I do quite miss this comment, as base_cont wasn't descriptive. [It could be worthwhile to specify why the last element is the base container name]
|
|
||
| # Adding 'docker://' to the container indicates this is a Docker image Singularity must convert | ||
| image_path = Client.pull('docker://' + container, name=sif_file) | ||
|
|
||
| # Check if the directory for base_cont already exists. When running concurrent jobs, it's possible | ||
| # something else has already pulled/unpacked the container. | ||
| # Here, we expand the sif image from `image_path` to a directory indicated by `base_cont` |
There was a problem hiding this comment.
(and these comments - the former comment for clarifying that it's necessary to hint to Apptainer that it is a docker container that should be converted)
No description provided.