Episode 3: Update the episode flow and narration. by TejeshPala · Pull Request #22 · carpentries-incubator/hpc-job-efficiency

TejeshPala · 2026-07-28T10:54:18Z

Addresses: #9

Addresses: carpentries-incubator#9 Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

github-actions · 2026-07-28T10:54:40Z

Thank you!

Thank you for your pull request 😃

🤖 This automated message can help you check the rendered files in your submission for clarity. If you have any questions, please feel free to open an issue in {sandpaper}.

If you have files that automatically render output (e.g. R Markdown), then you should check for the following:

🎯 correct output
🖼️ correct figures
❓ new warnings
‼️ new errors

Rendered Changes

🔍 Inspect the changes: https://github.com/carpentries-incubator/hpc-job-efficiency/compare/md-outputs..md-outputs-PR-22

The following changes were observed in the rendered markdown documents:

 03_schedulertools.md | 876 ++++++++++++++++++++++++++++++++-------------------
 md5sum.txt           |   2 +-
 2 files changed, 553 insertions(+), 325 deletions(-)

What does this mean?

If you have source files that require output and figures to be generated (e.g. R Markdown), then it is important to make sure the generated figures and output are reproducible.

This output provides a way for you to inspect the output in a diff-friendly manner so that it's easy to see the changes that occur due to new software versions or randomisation.

⏱️ Updated at 2026-07-28 23:46:25 +0000

Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

stderr-enst · 2026-07-29T07:06:29Z

Hey Tejesh,

thanks for the PR. I think introducing the scheduler in the beginning of the episode is too much.
We already rely on this info from the very first episode, which is why hpc-intro is a prerequisite. So the learners should know about Slurm already and the purpose of the episode here is to show what tools Slurm provide to understand job allocation.

I'd keep the intro to a brief paragraph, as before, and then go right into the submission of a short test job and into explaining sacct and seff in terms of resource utilization.

I think we also need a couple of callouts to note that Slurm sometimes does not collect the full information, e.g. on Energy (has Slurm even been configured to use RAPL counters? What information is not collected into the energy value?) and Disk I/O (local disk only? What parallel FS are supported and has Slurm been configured to track it properly?)

We should make sure that the duplication of seff with episode 02 is treated well. I think seff is useful in episode 02 (see #20 ), so maybe this means we don't need to spend too much time on it here? Are there other things we want to highlight about seff?

stderr-enst

Thanks again for the iteration Tejesh!
Here are a couple of in-line comments

stderr-enst · 2026-07-29T07:07:11Z

 ::::::::::::::::::::::::::::::::::::: objectives

-After completing this episode, participants should be able to …
+After completing this episode, participants should be able to:


nitpick: I think the dots are used in the other episodes as well, so we should keep it the same or use colons everywhere else.

stderr-enst · 2026-07-29T07:07:46Z

+In the previous episode, we learned how to choose appropriate resources for our jobs and why
+requesting them efficiently is important. Once a job has run on an HPC system, an important 
+question remains: Did it use the allocated resources efficiently? The scheduler provides tools
+that let us inspect completed jobs and answer that question.


I like the introductory paragraph!

stderr-enst · 2026-07-29T07:08:31Z

+## What is a Scheduler?

-A scheduler performs important tasks such as accepting and scheduling jobs,
-monitoring job status, starting user applications, cleaning up jobs that
-have finished or exceeded their allocated time. The scheduler also keeps
-a history of jobs that have been run and how they behaved. The information
-that is collected can be queried by the job owner to learn about how
-the job utilized the resources it was given.
+HPC systems are shared by many users. Every user wants to execute jobs, but
+CPUs, GPUs, memory, and compute nodes are limited resources. A **scheduler** is
+responsible for deciding when jobs run, where they run, and which resources they
+receive. It also monitors running jobs, records accounting information about their
+execution, and releases resources after they finish. This accounting information
+can later be queried by the job owner to understand the job's performance and
+resource utilization.

-### The `seff` tool
+### Why is a scheduler required?

-The `seff` command can be used to learn about how efficiently your job
-has run. The `seff` command takes the job identifier as an argument
-to select which job it displays information about. That means we need
-to run a job first to get a job identifier we can query SLURM about.
-Then we can ask about the efficiency of the job.
+Imagine fifty researchers all submit jobs to the same HPC cluster at the same time. Some jobs
+require only a few CPU cores for a few minutes, while other jobs request hundreds of cores for
+several hours. Since the available hardware is limited, not every job can start immediately.
+Without a scheduler, users would have to compete for resources manually, leading to conflicts,
+idle hardware, and unfair resource allocation.

-::: callout
+The scheduler coordinates all submitted jobs so that shared HPC resources are allocated
+fairly and efficiently.

-# `seff` may not be available
+### What tools does the scheduler provide?

-`seff` is an optional SLURM tool for more convenient access to `saact`. It does not come standard with every SLURM installation.
-Your particular HPC system may or may not provide it. Check for it's availability on your login nodes, or consult your cluster documentation or support staff.
+| Purpose | Example tools |
+| ------- | ------------- |
+| Submit batch jobs | `sbatch` |
+| Run interactive or parallel jobs | `srun`, `salloc` |
+| Monitor queued and running jobs | `squeue`, `sstat` |
+| View cluster information (nodes and partitions) | `sinfo` |
+| Cancel jobs | `scancel` |
+| Inspect completed jobs | `sacct`, `seff` |
+| ... | ... |

-Other third party alternatives, e.g. [reportseff](https://github.com/troycomi/reportseff/), can be installed with default user permissions.
+Schedulers provide many tools for submitting, monitoring, and managing jobs. In this episode,
+we focus only on the tools for analyzing completed jobs.

-:::::::::::
+These tools use the scheduler's accounting information to help us answer questions such as:
+
+- How long did the job actually run?
+- Did I request more time or resources than necessary?
+- Did my job use the allocated CPU resources efficiently?
+- Did I request enough memory?

-The `sbatch` command is used to submit a job. It takes a job script as
-an argument. The job script contains the resource requests, such as the
-amount of time needed for the calculation, the number of nodes, the
-number of tasks per node, and so on. It also contains the commands to
-execute the calculations.
+Throughout this episode, we will learn how to answer these questions using `sacct` and `seff`.
+
+### Submitting a job: `sbatch`
+
+We'll use `sbatch` once more to submit a job that we can later analyze. It takes
+a job script as an argument. The job script contains the resource requests, such
+as the amount of time needed for the calculation, the number of nodes, the number
+of tasks per node, and so on. It also contains the commands to execute the calculations.


We are already requiring the knowledge I think, as mentioned in the other comment.

stderr-enst · 2026-07-29T07:09:01Z

+### Monitoring jobs with `squeue`

 ```bash
-jobid=$(sbatch --parsable render_snowman.sbatch)
-seff $jobid
+# monitor for the user
+squeue --me
+# or squeue -u $USER
 ```

 ```output
-Job ID: 309489
-Cluster: bigiron
-User/Group: usr123/grp123
-State: COMPLETED (exit code 0)
-Nodes: 1
-Cores per node: 4
-CPU Utilized: 00:07:43
-CPU Efficiency: 98.93% of 00:07:48 core-walltime
-Job Wall-clock time: 00:01:57
-Memory Utilized: 35.75 MB
-Memory Efficiency: 0.20% of 17.58 GB (4.39 GB/core)
+             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
+          21271983     short render-s     user  R       0:04      1 wn21249


Basic, but we can keep it, just as a short repetition.

stderr-enst · 2026-07-29T07:09:22Z

+Periodically check the job status using `squeue --me`. Once the job is no longer listed,
+continue to the next section.
+
+| SLURM Status (ST) | Meaning |
+| ----------------- | ------- |
+| CA | Cancelled |
+| CD | Completed |
+| CF | Configuring |
+| CG | Completing |
+| F | Failed |
+| OOM | Out of Memory |
+| PD | Pending |
+| R | Running |
+| ST | Stopped |
+| S | Suspended |
+| TO | Timeout |
+| ... | ... |


I'd not go into the job states here, as mentioned above.

stderr-enst · 2026-07-29T07:13:47Z

+- `MaxRSS` and `AveRSS` report the maximum and average Resident Size Set (RSS), which
+  is the amount of memory actively resident in physical RAM. Comparing these values
+  with the memory requested for the job helps determine whether the memory request
+  is appropriate.


This is also addressed in detail Episode 02 (see the open Review PR for the latest version).
Slurm is measuring the memory utilization in regular intervals and may miss peaks. Probably good to keep the section here, but to also briefly refer on the section in episode 02, just as a repetition / connection.

stderr-enst · 2026-07-29T07:14:35Z

+### How much data moved to and from storage?
+
+- `MaxDiskRead` and `MaxDiskWrite` report the amount of data read from and written to
+  storage during the job. Disk activity may include not only application input and
+  output files but also reading the executable and shared libraries required to start
+  the program.
+
+### How much energy did the job consume?
+
+- `ConsumedEnergy` reports the energy used by the job if the HPC system is configured
+  to collect this information. On systems where energy accounting is not enabled, this
+  value is typically reported as zero.


These are likely dependent on the config and may leave out important info (Energy not tracking everything and Disk I/O not tracking parallel FS). So these need a callout/warning of some kind I think

stderr-enst · 2026-07-29T07:16:09Z

@@ -608,13 +565,284 @@ is responsible for the majority of data written.



Yeah, this is probably too close to the scaling study in Episode 04.
Maybe it's still a good exercise and why not do the scaling here as well with a brief look at it via sacct and seff?

But maybe keep it to 4 and 8 cores (and maybe 16) and just do a simple comparison and not going too much into the topic of "scaling".
The larger scaling study is a topic for episode 4

stderr-enst · 2026-07-29T07:16:57Z

+```output
+Job ID: 21271983
+Cluster: pleiades
+User/Group: <user>/<usrgrp>
+State: COMPLETED (exit code 0)
+Nodes: 1
+Cores per node: 4
+CPU Utilized: 00:02:02
+CPU Efficiency: 84.72% of 00:02:24 core-walltime
+Job Wall-clock time: 00:00:36
+Memory Utilized: 602.19 MB
+Memory Efficiency: 3.96% of 14.84 GB
+```
+
+The report combines several accounting fields into a concise overview of the resources
+allocated to the job and how effectively they were used. Rather than inspecting
+individual accounting fields, we can quickly identify whether the requested wall-clock
+time and memory were appropriate.
+
+The report summarizes three types of information:
+
+- Resources allocated
+    - Number of nodes
+    - Cores per node
+    - Allocated memory
+
+- Resources used
+    - CPU utilized
+    - Job wall-clock time
+    - Memory utilized
+
+- Efficiency
+    - CPU Efficiency
+    - Memory Efficiency


It's good to introduce the complete output of seff! We don't do that in Episode 02. But we should make sure there is no redundancy between the two.

stderr-enst · 2026-07-29T07:19:12Z

+:::::::::::::::::::: challenge
+
+# Does the memory usage efficiency improve?
+
+Edit the batch file to reduce the amount of memory requested for the
+job. Note that the amount of memory requested per CPU core can be specified with the
+`--mem-per-cpu=` option.
+
+Memory amounts may be specified using the following suffixes:
+
+- `K` or `k` for kilobytes
+- `M` or `m` for megabytes
+- `G` or `g` for gigabytes
+- `T` or `t` for terabytes
+
+For this example, requesting 160 MB per CPU core is more than sufficient. Submit the job,
+and inspect the efficiency with `seff`. How does the memory efficiency compare with the
+previous submission?


This is a good exercise, but we do something similar in Episode 02, since it goes into the details of how Slurm tracks peak memory. So we shouldn't explain too much that already has been covered before and this could be just another way of looking at the same issue?

Update the episode flow and narration.

4a0060b

Addresses: carpentries-incubator#9 Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

accounting narration and flow correction

1324781

Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

github-actions Bot pushed a commit that referenced this pull request Jul 28, 2026

differences for PR #22

0a05754

stderr-enst changed the title ~~Update the episode flow and narration.~~ Episode 3: Update the episode flow and narration. Jul 28, 2026

corrections to seff interpretation and challenges

608d01a

Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

github-actions Bot pushed a commit that referenced this pull request Jul 28, 2026

differences for PR #22

1c1a6a3

move the accounting challenge to the correct place

5a3d721

Signed-off-by: TEJESH PALA <tpala@uni-wuppertal.de>

github-actions Bot pushed a commit that referenced this pull request Jul 28, 2026

differences for PR #22

60d5f17

stderr-enst reviewed Jul 29, 2026

View reviewed changes

		@@ -608,13 +565,284 @@ is responsible for the majority of data written.

Uh oh!

Uh oh!

Conversation

TejeshPala commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Thank you!

Rendered Changes

Uh oh!

stderr-enst commented Jul 29, 2026

Uh oh!

stderr-enst left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 28, 2026 •

edited

Loading