November 8, 2026 · Temple Compute
How to Right-Size Slurm Jobs (and Stop Wasting Hours)
Every HPC cluster has the same hidden tax. Somebody copied an #SBATCH
header years ago, padded it "to be safe", and it has been copied forward
ever since. The jobs run, so nobody looks. Meanwhile they reserve cores and
memory they never touch, wait longer in the queue, and burn fairshare or
cloud budget for nothing.
How do you right-size a Slurm job?
You right-size a Slurm job by measuring what a representative run actually
used, then requesting that plus modest headroom. Check CPU efficiency and
peak memory with seff or sacct, set --cpus-per-task to the cores the
job really kept busy, and set --mem to about 1.5 times the measured peak.
Repeat per step, not per pipeline.
Here is what that looks like on real data. These rows come from our nf-core/rnaseq benchmark on an AWS Slurm cluster, comparing nf-core's default resource labels with what each task measured:
| Task | Requested (cpus / GB) | Measured (cores / peak GB) | Right-sized (cpus / GB) |
|---|---|---|---|
| salmon_strand | 6 / 36 | 0.63 / 0.07 | 1 / 2 |
| stringtie | 6 / 36 | 1.44 / 0.96 | 1 / 2 |
| bedClip (clip_fw) | 6 / 36 | 0.95 / 0.06 | 1 / 2 |
| markdup | 6 / 36 | 1.07 / 19.08 | 1 / 29 |
| salmon_quant | 6 / 36 | 5.55 / 8.64 | 6 / 13 |
| star_align | 8 / 58 | 4.27 / 35.29 | 8 / 58 (kept) |
Across the whole run, the defaults requested 65 to 67 CPU-hours while the jobs used about 29. That is roughly 2.2 times more CPU reserved than consumed.
Step 1: read one job with seff
seff is the fastest way to see whether a single job was sized well. It
ships with Slurm's contrib tools and is installed on most clusters:
seff 123456
An illustrative output for an oversized job:
Job ID: 123456
State: COMPLETED (exit code 0)
Cores per node: 16
CPU Utilized: 02:10:33
CPU Efficiency: 13.60% of 16:00:00 core-walltime
Job Wall-clock time: 01:00:00
Memory Utilized: 3.20 GB
Memory Efficiency: 5.00% of 64.00 GB
Read it in two lines. CPU efficiency of 13.6% on 16 cores means the job kept
about two cores busy, so --cpus-per-task=2 or 3 is the honest request.
Memory efficiency of 5% of 64 GB means a 3.2 GB peak, so --mem=5G covers
it with headroom. The same work, one fifth of the reservation.
Step 2: compare many jobs with sacct
One job is an anecdote. sacct shows the pattern across every run of a
step, which is what you size against:
sacct -u $USER -S 2026-10-01 -P \
--format=JobID,JobName,ReqCPUS,AllocCPUS,TotalCPU,Elapsed,ReqMem,MaxRSS,State
The two efficiency formulas:
CPU efficiency = TotalCPU / (AllocCPUS × Elapsed)
Memory efficiency = MaxRSS / ReqMem
MaxRSS is reported per job step, so read the .batch or named step line,
not only the parent job line. Group the output by JobName and look at the
spread: if one step's memory varies with input size, size for the large
inputs, not the average. The
Minnesota Supercomputing Institute's job sizing guide
suggests aiming for around 80 to 90% efficiency.
Anything under 50% is reserved capacity that does nothing.
Step 3: apply a sizing rule
A sizing rule you apply the same way every time beats intuition. This is the rule we used to size all 56 task types in our port of nf-core/rnaseq:
- CPUs = mean cores actually used, rounded, at least 1.
- Memory = 1.5 × measured peak, rounded up, at least 2 GB.
- Cap at one node. If a step needs more, it is a different problem.
Then check for tools whose thread flags are tied to the request. In the
benchmark we kept STAR at 8 cpus and 58 GB: at a 35 GB peak only one STAR
fits on the high-memory node anyway. We kept Trim Galore at 8 cpus because
its --cores value is derived from the CPU count, and cutting it would have
cut its parallelism. Every other thread and memory flag was derived from the
task's resources, so the tools used exactly what they were given.
Step 4: GPU jobs need their own measurement
GPU jobs are where oversizing gets expensive. Slurm accounting reliably
records which GPUs a job was allocated, but on many sites it does not record
how busy they were. seff will tell you the CPU side and nothing useful
about the GPU.
Log the GPU alongside the job instead:
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used \
--format=csv -l 5 > gpu_usage.csv &
Two questions matter. Is the GPU busy most of the run? If utilization hovers at 20%, the job is bound by data loading or CPU work. And how many CPU cores does a GPU job really need? For GPU molecular dynamics the answer is usually very few. When a group running AMBER could finally see that, their CPU core requests dropped by 76% with the same simulations on the same GPUs.
What right-sizing bought in our benchmark
Right-sizing is not tidiness. It is the single biggest performance lever most pipelines leave untouched. On nf-core/rnaseq with 12 human samples on an AWS Slurm cluster, a right-sized run of Horus, by Temple Compute, finished 35% faster and 32% cheaper than Nextflow with nf-core's default resources.
Holding everything else equal, the Horus run with nf-core's default sizes against the right-sized Horus run shows where that came from:
| Same Horus pipeline | Default sizes | Right-sized |
|---|---|---|
| CPU-hours requested | 62.2 | 47.4 |
| CPU-hours used | 28.1 | 30.0 |
| Mean queue wait per job | 468 s | 53 s |
| Wall time | 8,210 s | 5,739 s |
| Run cost | $8.83 | $6.38 |
The work done was the same. Smaller requests let more tasks share each node, so tasks stopped waiting for a whole node to free up, and the pipeline finished 30% sooner. Queue time, not compute time, was the bottleneck.
Why most pipelines never get right-sized
Most pipelines never get right-sized because the measurement is too
tedious. Slurm records usage one job at a time, after each job ends. Our
nf-core/rnaseq run over 12 samples was about 500 Slurm jobs. Nobody runs seff 500
times, so the padded defaults survive indefinitely.
Horus measures every task as part of the run. Its resource monitor records CPU, peak memory and wall time for each command, plus GPU memory and utilization where present, and reads them back over the same SSH connection it uses to run the job. At the end of a run every task sits in one table and one JSON report. In the benchmark, that single report from one run, turned into sizes by a script, was all the input needed to generate the right-sized configuration. You still choose the numbers. Horus makes the evidence impossible to miss.
Provision the right infrastructure, not the biggest
Right-sizing does not stop at the job request. It decides which machines you need at all. Once each step's real footprint is known, pick the smallest node or cloud instance that fits it, and let small steps pack onto shared nodes instead of claiming a large one each.
With Horus, each workflow stage declares where it may run. Steps pinned to your own cluster stay there, on hours you already own. Stages you allow to run in the cloud get an on-demand machine sized to the stage, created when the run reaches it and destroyed when the run ends. No reserved fleet, no idle nodes overnight.
Where to go next
If you manage the cluster or the budget rather than the jobs, read research computing cost control for budgets and limits that enforce this at the organization level. If you chain many jobs together, a Slurm workflow manager handles dependencies, retries and per-step resources for you. For sweeps over many parameters, the same sizing logic applies to parametric sweeps. The engine itself is open source: see Horus.
FAQ
How do I check how much CPU and memory a Slurm job actually used?
Run seff with the job ID for a one-job summary of CPU and memory efficiency. For many jobs, use sacct with a format such as JobID,ReqCPUS,AllocCPUS,TotalCPU,Elapsed,ReqMem,MaxRSS and compare what was requested with what was used.
What is a good CPU efficiency for a Slurm job?
HPC centres such as the Minnesota Supercomputing Institute recommend aiming for high efficiency, around 80 to 90 percent. Below roughly 50 percent you are holding cores that do nothing, which costs fairshare on a shared cluster and money on a billed or cloud cluster.
How much memory should I request for a Slurm job?
Measure the peak resident memory of a representative run, then request about 1.5 times that peak with a sensible floor such as 2 GB. That leaves headroom for input variation without reserving memory the job never touches.
Do smaller Slurm jobs start sooner?
Usually. Smaller requests fit into scheduling gaps and pack more jobs per node. In our nf-core/rnaseq benchmark, right-sizing cut the mean queue wait per job from 468 seconds to 53 seconds on the same cluster.
Can Slurm show GPU utilization?
Slurm accounting reliably records which GPUs a job was allocated, but on many sites it does not record how busy they were. Log nvidia-smi alongside the job, or use a tool that samples GPU utilization and memory per task.