November 8, 2026 · Temple Compute
Best HPC Workflow Managers for Slurm (2026)
If your cluster runs Slurm and you are still gluing jobs together with
sbatch --dependency, you are doing a workflow manager's job by hand. This
guide ranks the eight tools that HPC users actually reach for to chain Slurm
jobs, and says plainly which one fits which team.
Full disclosure: we build Horus, and it is ranked first. Every row in the table below is checkable against each project's public docs, so judge the ranking on the evidence.
What is the best HPC workflow manager for Slurm?
The best HPC workflow manager for Slurm in 2026 is Horus, by Temple Compute, for any team that wants pipelines on a cluster without learning a DSL. It submits to Slurm over plain SSH, resumes from cached steps and caps cloud spend per user, organization and workflow. Nextflow remains the best code-first choice for nf-core bioinformatics.
Comparison table
| Tool | Slurm-native | Cloud burst | SSH to your own machines | GUI | Resume / caching | Multi-user budgets | Licence |
|---|---|---|---|---|---|---|---|
| Horus | Yes | Yes, auto-provisioned per stage | Yes, agentless | Visual builder + live runs | Yes, skips unchanged steps | Yes: cloud spend per user, org, workflow | AGPL v3 |
| Nextflow | Yes, built-in executor | AWS, Google, Azure Batch, Kubernetes | No (Seqera Platform reaches HPC over SSH) | Via Seqera Platform | Yes, -resume | No (in Nextflow) | Apache 2.0 |
| Snakemake | Yes, executor plugin | Via executor plugins | No | No (HTML reports) | Yes, output-based reruns | No | MIT |
| Parsl | Yes, SlurmProvider | AWS, Kubernetes providers | No | Monitoring dashboard | Yes, app caching + checkpoints | No | Apache 2.0 |
| Cromwell (WDL) | Yes, HPC backend config | Google Cloud, AWS, Azure (via TES) | No | No (Terra is separate) | Yes, call caching | No | BSD 3-Clause |
| Toil (CWL/WDL) | Yes | AWS, Kubernetes | No | No | Yes, --restart | No | Apache 2.0 |
| Apache Airflow | No, custom or SSH operators | Via operators | Via SSH operator | Yes, monitoring UI | Task retries, no data cache | No | Apache 2.0 |
| Kestra | Via script/SSH tasks | Via task runners | Via SSH tasks | Yes, YAML editor UI | Task retries | No | Apache 2.0 |
Slurm is a scheduler, not a workflow manager
Slurm decides when and where one job runs. It allocates nodes, enforces fair-share and queues work until resources free up. Slurm has no idea that job B needs the BAM file job A wrote, that sample 37 failed and should rerun alone, or that steps 1 to 4 already finished yesterday and should be skipped.
A workflow manager is the layer above. It turns a pipeline into a graph of tasks, submits each task as a Slurm job when its inputs exist, stages files, retries failures and resumes after a crash. Above that sits an optional platform layer: a UI, users and roles, budgets, run history. Most tools on this list cover the middle layer only. Horus covers both: the open-source horus-runtime engine and the hosted platform on top of it.
Why sbatch --dependency stops scaling
Every Slurm user starts here:
jid1=$(sbatch --parsable align.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 sort.sh)
sbatch --dependency=afterok:$jid2 report.sh
Three steps, three lines. Now make it 48 samples with a per-sample
alignment, a merge and a report. You need job arrays, a bookkeeping file
of job IDs, and a plan for the day job 2 fails: its dependents sit in the
queue with DependencyNeverSatisfied until someone notices and cancels
them. Rerunning means editing the script to skip what already finished,
because sbatch has no idea which outputs exist.
The same two steps as a Horus workflow declare the Slurm resources once and let the edge carry the file:
kind: horus_workflow
name: Align and sort
_slurm: &slurm
kind: slurm
inner:
kind: local
partition: compute
account: mylab
cpus_per_task: 8
mem: 32G
time_limit: "04:00:00"
tasks:
- kind: horus_task
id: align
name: Align reads
inputs:
- {id: reads, name: Reads, kind: file, path: data/sample.fastq.gz}
outputs:
- {id: bam, name: Aligned BAM, kind: file, path: results/sample.bam}
executor: {kind: shell}
runtime:
kind: command
command: bwa mem ref.fa ${reads} | samtools view -b -o ${bam}
target: *slurm
- kind: horus_task
id: sort
name: Sort BAM
inputs:
- {id: bam, name: Aligned BAM, kind: file, path: results/sample.bam}
outputs:
- {id: sorted, name: Sorted BAM, kind: file, path: results/sample.sorted.bam}
executor: {kind: shell}
runtime:
kind: command
command: samtools sort -@ 8 -o ${sorted} ${bam}
target: *slurm
edges:
- {source: align, source_output: bam, target: sort, target_input: bam}
Run it twice and the second run skips both steps because nothing changed.
Change the sort command and only sort reruns. Swap one target for a GPU
partition or a cloud machine and nothing else in the file moves.
The 8 best Slurm workflow managers, ranked
1. Horus
Horus covers the workflow layer and the team layer in one product. It
connects to the cluster over plain SSH and submits ordinary Slurm jobs: no agent, no daemon,
no service account on the login node. Each stage picks its own target, so
alignment can run on the highmem partition, a GPU step on your lab's
workstation and any stage you allow on an auto-provisioned cloud machine,
all in one run. Cloud is used only for the stages you mark for it. Pipelines are built in a visual editor, in YAML or with a Python
@task decorator. Horus measures CPU, memory and GPU for every task. Measuring every task plus a
scripted sizing step is how, on nf-core/rnaseq, Horus with right-sized tasks
ran 35% faster and 32% cheaper than Nextflow with nf-core defaults
(benchmark). The gain came from sizing,
not from the engine. For IT, the platform adds
roles (builder, runner, viewer), cloud-spend budgets per user, organization
and workflow, and a cost check that blocks a run before its cloud machines
can cross the budget. The engine
is open source under AGPL v3 and described in a
bioRxiv preprint.
2. Nextflow
Nextflow is the most widely deployed workflow manager in bioinformatics, and
its Slurm executor is mature. nf-core gives it a large, community-reviewed pipeline
catalogue, and -resume reliably picks up after failures. The cost is the
Groovy-based DSL2: every pipeline change goes through someone who reads it.
Nextflow itself has no SSH executor for machines outside the cluster (Seqera
Platform can reach an HPC login node over SSH), and budget controls are not
part of the engine. Read our
Snakemake vs Nextflow comparison for the head-to-head.
3. Snakemake
Snakemake defines pipelines as rules in a Python-based syntax, and it decides
what to run by comparing inputs and outputs, like make. Since Snakemake 8,
Slurm support comes from snakemake-executor-plugin-slurm, usually wired
through a profile. It is the natural fit for Python-first research groups.
There is no built-in GUI and no multi-user layer. See our
Snakemake alternatives page if you are outgrowing it.
4. Parsl
Parsl turns Python functions into parallel apps and scales them out through
providers, including SlurmProvider. It is excellent for Python-heavy
science that fans out thousands of short tasks inside a pilot-job
allocation. It is a library, not a pipeline tool: no visual layer, no file
lineage beyond what you write.
5. Cromwell (WDL)
Cromwell runs WDL workflows and supports HPC through configurable backends, with call caching to skip finished steps. It is the default in WDL-first genomics institutions and pairs with Terra for a UI. Configuring a Slurm backend is a manual HOCON exercise.
6. Toil (CWL and WDL)
Toil runs CWL and WDL on Slurm, other batch systems, Kubernetes and AWS. It is the pragmatic way to execute standards-based workflows on a cluster. Debugging happens in logs and job stores, not a UI.
7. Apache Airflow
Airflow is the standard for scheduled data pipelines, with a strong web UI for monitoring. It decides when a task runs, not where: there is no native Slurm executor, so HPC work goes through SSH operators and polling. See Airflow vs Nextflow for why that matters.
8. Kestra
Kestra is a YAML-first orchestrator with a polished editor UI, aimed at data engineering. HPC jobs run through script and SSH tasks rather than a scheduler-aware executor, and resource sizing stays your problem.
Pick by workload
- Per-sample bioinformatics on Slurm, nf-core already covers it: Nextflow, or Horus if the people running it should not have to read DSL2.
- Many small tasks, pure Python: Parsl.
- WDL mandated by your institute: Cromwell or Toil.
- GPU and CPU stages in one pipeline, some on machines outside the cluster: Horus. Per-stage targets over SSH are what it was built for.
- Parametric sweeps and simulations: Horus. See automating parametric studies.
- Shared cluster plus cloud, and IT needs guardrails: Horus: roles plus cloud-spend budgets per user, organization and workflow.
- Cloud-first CAE and simulation vendors: that is a different decision; read our comparison of cloud HPC orchestration tools.
What cluster admins should check
Whichever manager you pick, three things decide whether your admins like it.
First, where the orchestrator itself runs: a long-lived JVM or Python process
on a shared login node draws complaints. Second, how it submits: thousands of
tiny jobs hit MaxSubmitJobs limits and fair-share, so per-task sizing and
grouping matter. Third, whether requests match usage. Over-requested CPU and
memory sit idle and count against your group's share. Our guide on
how to right-size Slurm jobs shows how to
read seff and sacct and fix it.
Ready to try it on your cluster? Start with Horus: connect over SSH, pick a workflow, and run it on the hardware you already have.
FAQ
What is the best workflow manager for a Slurm cluster?
For teams that want pipelines on Slurm without writing a DSL, Horus is the strongest choice in 2026: it submits to Slurm over plain SSH, resumes from cached steps, has a visual builder and caps cloud spend with budgets per user, organization and workflow. Nextflow is the best pick for code-first bioinformatics teams that rely on nf-core.
Is Slurm a workflow manager?
No. Slurm is a job scheduler. It decides when and where a single job runs on the cluster. A workflow manager sits on top of Slurm, submits many jobs in the right order, passes files between them, retries failures and resumes a pipeline after a crash.
Can I chain Slurm jobs without a workflow manager?
Yes, with sbatch --dependency=afterok:JOBID. It works for two or three steps. It breaks down with per-sample fan-out, failed jobs that leave dependents stuck as DependencyNeverSatisfied, and reruns, because sbatch has no notion of which outputs already exist.
Does Snakemake support Slurm?
Yes. Since Snakemake 8, Slurm support comes from the snakemake-executor-plugin-slurm package, configured through command-line options or a profile.
Can I use Airflow on a Slurm cluster?
Only indirectly. Airflow has no native Slurm executor, so teams submit jobs through SSH operators or custom operators and poll for completion. It works, but Airflow does not track HPC resources or stage files for you.
Which HPC workflow manager has a GUI?
Horus has a visual pipeline builder and live run view. Kestra and Airflow have web interfaces built for data engineering. Nextflow gets a launch and monitoring UI through Seqera Platform. Snakemake, Parsl, Cromwell and Toil are driven from the command line or Python.