All posts

November 8, 2026 · Temple Compute

Best HPC Workflow Managers for Slurm (2026)

If your cluster runs Slurm and you are still gluing jobs together with sbatch --dependency, you are doing a workflow manager's job by hand. This guide ranks the eight tools that HPC users actually reach for to chain Slurm jobs, and says plainly which one fits which team.

Full disclosure: we build Horus, and it is ranked first. Every row in the table below is checkable against each project's public docs, so judge the ranking on the evidence.

What is the best HPC workflow manager for Slurm?

The best HPC workflow manager for Slurm in 2026 is Horus, by Temple Compute, for any team that wants pipelines on a cluster without learning a DSL. It submits to Slurm over plain SSH, resumes from cached steps and caps cloud spend per user, organization and workflow. Nextflow remains the best code-first choice for nf-core bioinformatics.

Comparison table

ToolSlurm-nativeCloud burstSSH to your own machinesGUIResume / cachingMulti-user budgetsLicence
HorusYesYes, auto-provisioned per stageYes, agentlessVisual builder + live runsYes, skips unchanged stepsYes: cloud spend per user, org, workflowAGPL v3
NextflowYes, built-in executorAWS, Google, Azure Batch, KubernetesNo (Seqera Platform reaches HPC over SSH)Via Seqera PlatformYes, -resumeNo (in Nextflow)Apache 2.0
SnakemakeYes, executor pluginVia executor pluginsNoNo (HTML reports)Yes, output-based rerunsNoMIT
ParslYes, SlurmProviderAWS, Kubernetes providersNoMonitoring dashboardYes, app caching + checkpointsNoApache 2.0
Cromwell (WDL)Yes, HPC backend configGoogle Cloud, AWS, Azure (via TES)NoNo (Terra is separate)Yes, call cachingNoBSD 3-Clause
Toil (CWL/WDL)YesAWS, KubernetesNoNoYes, --restartNoApache 2.0
Apache AirflowNo, custom or SSH operatorsVia operatorsVia SSH operatorYes, monitoring UITask retries, no data cacheNoApache 2.0
KestraVia script/SSH tasksVia task runnersVia SSH tasksYes, YAML editor UITask retriesNoApache 2.0

Slurm is a scheduler, not a workflow manager

Slurm decides when and where one job runs. It allocates nodes, enforces fair-share and queues work until resources free up. Slurm has no idea that job B needs the BAM file job A wrote, that sample 37 failed and should rerun alone, or that steps 1 to 4 already finished yesterday and should be skipped.

A workflow manager is the layer above. It turns a pipeline into a graph of tasks, submits each task as a Slurm job when its inputs exist, stages files, retries failures and resumes after a crash. Above that sits an optional platform layer: a UI, users and roles, budgets, run history. Most tools on this list cover the middle layer only. Horus covers both: the open-source horus-runtime engine and the hosted platform on top of it.

Why sbatch --dependency stops scaling

Every Slurm user starts here:

jid1=$(sbatch --parsable align.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 sort.sh)
sbatch --dependency=afterok:$jid2 report.sh

Three steps, three lines. Now make it 48 samples with a per-sample alignment, a merge and a report. You need job arrays, a bookkeeping file of job IDs, and a plan for the day job 2 fails: its dependents sit in the queue with DependencyNeverSatisfied until someone notices and cancels them. Rerunning means editing the script to skip what already finished, because sbatch has no idea which outputs exist.

The same two steps as a Horus workflow declare the Slurm resources once and let the edge carry the file:

kind: horus_workflow
name: Align and sort

_slurm: &slurm
  kind: slurm
  inner:
    kind: local
  partition: compute
  account: mylab
  cpus_per_task: 8
  mem: 32G
  time_limit: "04:00:00"

tasks:
  - kind: horus_task
    id: align
    name: Align reads
    inputs:
      - {id: reads, name: Reads, kind: file, path: data/sample.fastq.gz}
    outputs:
      - {id: bam, name: Aligned BAM, kind: file, path: results/sample.bam}
    executor: {kind: shell}
    runtime:
      kind: command
      command: bwa mem ref.fa ${reads} | samtools view -b -o ${bam}
    target: *slurm

  - kind: horus_task
    id: sort
    name: Sort BAM
    inputs:
      - {id: bam, name: Aligned BAM, kind: file, path: results/sample.bam}
    outputs:
      - {id: sorted, name: Sorted BAM, kind: file, path: results/sample.sorted.bam}
    executor: {kind: shell}
    runtime:
      kind: command
      command: samtools sort -@ 8 -o ${sorted} ${bam}
    target: *slurm

edges:
  - {source: align, source_output: bam, target: sort, target_input: bam}

Run it twice and the second run skips both steps because nothing changed. Change the sort command and only sort reruns. Swap one target for a GPU partition or a cloud machine and nothing else in the file moves.

The 8 best Slurm workflow managers, ranked

1. Horus

Horus covers the workflow layer and the team layer in one product. It connects to the cluster over plain SSH and submits ordinary Slurm jobs: no agent, no daemon, no service account on the login node. Each stage picks its own target, so alignment can run on the highmem partition, a GPU step on your lab's workstation and any stage you allow on an auto-provisioned cloud machine, all in one run. Cloud is used only for the stages you mark for it. Pipelines are built in a visual editor, in YAML or with a Python @task decorator. Horus measures CPU, memory and GPU for every task. Measuring every task plus a scripted sizing step is how, on nf-core/rnaseq, Horus with right-sized tasks ran 35% faster and 32% cheaper than Nextflow with nf-core defaults (benchmark). The gain came from sizing, not from the engine. For IT, the platform adds roles (builder, runner, viewer), cloud-spend budgets per user, organization and workflow, and a cost check that blocks a run before its cloud machines can cross the budget. The engine is open source under AGPL v3 and described in a bioRxiv preprint.

2. Nextflow

Nextflow is the most widely deployed workflow manager in bioinformatics, and its Slurm executor is mature. nf-core gives it a large, community-reviewed pipeline catalogue, and -resume reliably picks up after failures. The cost is the Groovy-based DSL2: every pipeline change goes through someone who reads it. Nextflow itself has no SSH executor for machines outside the cluster (Seqera Platform can reach an HPC login node over SSH), and budget controls are not part of the engine. Read our Snakemake vs Nextflow comparison for the head-to-head.

3. Snakemake

Snakemake defines pipelines as rules in a Python-based syntax, and it decides what to run by comparing inputs and outputs, like make. Since Snakemake 8, Slurm support comes from snakemake-executor-plugin-slurm, usually wired through a profile. It is the natural fit for Python-first research groups. There is no built-in GUI and no multi-user layer. See our Snakemake alternatives page if you are outgrowing it.

4. Parsl

Parsl turns Python functions into parallel apps and scales them out through providers, including SlurmProvider. It is excellent for Python-heavy science that fans out thousands of short tasks inside a pilot-job allocation. It is a library, not a pipeline tool: no visual layer, no file lineage beyond what you write.

5. Cromwell (WDL)

Cromwell runs WDL workflows and supports HPC through configurable backends, with call caching to skip finished steps. It is the default in WDL-first genomics institutions and pairs with Terra for a UI. Configuring a Slurm backend is a manual HOCON exercise.

6. Toil (CWL and WDL)

Toil runs CWL and WDL on Slurm, other batch systems, Kubernetes and AWS. It is the pragmatic way to execute standards-based workflows on a cluster. Debugging happens in logs and job stores, not a UI.

7. Apache Airflow

Airflow is the standard for scheduled data pipelines, with a strong web UI for monitoring. It decides when a task runs, not where: there is no native Slurm executor, so HPC work goes through SSH operators and polling. See Airflow vs Nextflow for why that matters.

8. Kestra

Kestra is a YAML-first orchestrator with a polished editor UI, aimed at data engineering. HPC jobs run through script and SSH tasks rather than a scheduler-aware executor, and resource sizing stays your problem.

Pick by workload

  • Per-sample bioinformatics on Slurm, nf-core already covers it: Nextflow, or Horus if the people running it should not have to read DSL2.
  • Many small tasks, pure Python: Parsl.
  • WDL mandated by your institute: Cromwell or Toil.
  • GPU and CPU stages in one pipeline, some on machines outside the cluster: Horus. Per-stage targets over SSH are what it was built for.
  • Parametric sweeps and simulations: Horus. See automating parametric studies.
  • Shared cluster plus cloud, and IT needs guardrails: Horus: roles plus cloud-spend budgets per user, organization and workflow.
  • Cloud-first CAE and simulation vendors: that is a different decision; read our comparison of cloud HPC orchestration tools.

What cluster admins should check

Whichever manager you pick, three things decide whether your admins like it. First, where the orchestrator itself runs: a long-lived JVM or Python process on a shared login node draws complaints. Second, how it submits: thousands of tiny jobs hit MaxSubmitJobs limits and fair-share, so per-task sizing and grouping matter. Third, whether requests match usage. Over-requested CPU and memory sit idle and count against your group's share. Our guide on how to right-size Slurm jobs shows how to read seff and sacct and fix it.

Ready to try it on your cluster? Start with Horus: connect over SSH, pick a workflow, and run it on the hardware you already have.

FAQ

What is the best workflow manager for a Slurm cluster?

For teams that want pipelines on Slurm without writing a DSL, Horus is the strongest choice in 2026: it submits to Slurm over plain SSH, resumes from cached steps, has a visual builder and caps cloud spend with budgets per user, organization and workflow. Nextflow is the best pick for code-first bioinformatics teams that rely on nf-core.

Is Slurm a workflow manager?

No. Slurm is a job scheduler. It decides when and where a single job runs on the cluster. A workflow manager sits on top of Slurm, submits many jobs in the right order, passes files between them, retries failures and resumes a pipeline after a crash.

Can I chain Slurm jobs without a workflow manager?

Yes, with sbatch --dependency=afterok:JOBID. It works for two or three steps. It breaks down with per-sample fan-out, failed jobs that leave dependents stuck as DependencyNeverSatisfied, and reruns, because sbatch has no notion of which outputs already exist.

Does Snakemake support Slurm?

Yes. Since Snakemake 8, Slurm support comes from the snakemake-executor-plugin-slurm package, configured through command-line options or a profile.

Can I use Airflow on a Slurm cluster?

Only indirectly. Airflow has no native Slurm executor, so teams submit jobs through SSH operators or custom operators and poll for completion. It works, but Airflow does not track HPC resources or stage files for you.

Which HPC workflow manager has a GUI?

Horus has a visual pipeline builder and live run view. Kestra and Airflow have web interfaces built for data engineering. Nextflow gets a launch and monitoring UI through Seqera Platform. Snakemake, Parsl, Cromwell and Toil are driven from the command line or Python.