Airflow alternative for HPC and GPUs: Horus vs Airflow

What Airflow is good at

Airflow is the default answer for scheduled data orchestration, and deservedly so. It has an enormous library of integrations, a mature scheduler, good observability into DAG runs, and more than a decade of production hardening. If you need to move data between warehouses on a cron schedule and know when a step failed at 3am, Airflow does that well and everybody you hire will already know it.

It is worth being precise about why it often gets considered for compute-heavy work: it is the orchestrator people already have. That is a real advantage, and sometimes the right one.

Where the two diverge

Airflow orchestrates. It does not provide compute.

This is the central difference, and it is architectural rather than a missing feature. Airflow decides when a task runs and in what order. Where it runs is entirely your problem: you bring a Kubernetes cluster, a Celery worker pool, an ECS cluster or a batch queue, and you keep it running.

For a data team with a platform group, that separation is a feature. For a research or engineering team without one, it means the hard part is untouched. The reason the team cannot run its pipeline is not that it lacks a scheduler; it is that provisioning a GPU node with the right software on it is a week of work.

Horus provisions the compute as part of running the workflow, runs each stage on the hardware it needs, and moves the files between stages itself.

It was designed to move data, not to run heavy compute

Airflow's model assumes tasks that are short, idempotent, and roughly uniform: extract, transform, load. Compute-heavy workloads break most of those assumptions. A simulation sweep fans out to hundreds of solver runs on a cluster. A training stage needs eight GPUs for a day, while the report after it needs a single core. Resource needs vary by orders of magnitude between stages in one pipeline. A molecular dynamics campaign in life sciences has the same shape.

You can express this in Airflow, and people do. It fights you: per-task resource isolation is awkward, long-running tasks sit badly with the scheduler's assumptions, and dynamic fan-out has improved but remains a rough edge.

No HPC schedulers

Airflow does not submit to SLURM, PBS or LSF. If your institution has a cluster, reaching it means writing and maintaining a custom operator that shells out to sbatch and then polls for completion, including all the failure handling. Many teams have written exactly this. It is nobody's favourite code.

Python and DAG authorship

Airflow DAGs are Python. As with Modal, this puts a developer between the pipeline and the researchers or engineers who own it, and every pipeline change is a code change, reviewed and deployed.

Looking at results

Airflow tells you whether a task succeeded. It has no opinion about what the task produced, which is correct for its purpose and unhelpful when the output is a results table, a trained model or a plot someone needs to look at.

Honest summary

	Apache Airflow	Horus
DAG definition	Python	Visual, YAML, or Python
Usable without writing code	No	Yes
Provides compute	No, you bring it	Yes, provisioned
HPC schedulers	Custom operators only	Native
Long-running simulation stages	Awkward	Designed for it
Per-stage resource isolation	Limited	Yes
Scheduled/cron orchestration	Excellent	Basic
Integration ecosystem	Enormous	Plugin catalogue
Domain result visualization	No	Yes

Airflow is better than us at scheduled data orchestration and has an ecosystem we will not match. Keep it for your data pipelines. Consider Horus for the compute-heavy stages alongside them (simulations, training, large batch jobs), where the problem is not sequencing tasks but getting hardware with the right software on it, without hiring someone to maintain a cluster.

Comparing the two directly? See Airflow vs Nextflow.