All posts

November 8, 2026 · Temple Compute

Automate NGS Data Processing for Sequencing Cores

What does automated NGS data processing look like?

Automated NGS data processing means a finished sequencing run turns into delivered, QC'd, analyzed results with no person submitting jobs. One pipeline detects the run, demultiplexes and QCs it, runs each client's assay-specific analysis on the cluster or cloud, delivers outputs and records cost. Horus, by Temple Compute, can run each stage you build into that chain.

Most sequencing facilities already have every piece. They have bcl-convert, FastQC and MultiQC, nf-core pipelines, a Slurm cluster and a LIMS. What they lack is the glue: today a staff bioinformatician notices the run finished, copies a samplesheet, launches the right pipeline for each project, checks it overnight, packages results and emails clients. That person is the throughput bottleneck, and they are expensive. This post lays out the reference architecture that removes them from the loop, and how it differs for three kinds of high-throughput data organizations.

The reference architecture: from instrument to invoice

A production NGS automation pipeline has six stages. Each one can be a stage in a single Horus workflow, so the run is one object with one history.

  1. Trigger. The instrument finishes and writes its completion marker. A small watcher script or your LIMS launches the workflow through the horus-runtime Python API or the Horus MCP server, passing the run folder and samplesheet.
  2. Demultiplex and QC. bcl-convert splits reads by sample; FastQC and MultiQC produce per-sample and per-run quality reports. Failing samples are flagged before anyone spends compute on them.
  3. Per-assay analysis. Each client's samples route to their pipeline: RNA-seq, variant calling, amplicon, single-cell. These fan out one task per sample on Slurm, or on cloud machines for the stages you allow.
  4. Delivery. Outputs are packaged per client with their QC report, and every artifact carries lineage back to the run, the pipeline version and the parameters.
  5. Retention. Raw and intermediate data follow your policy; the lineage record stays.
  6. Cost recovery. Compute cost is recorded for every run, and budgets per user, organization and workflow cap cloud spend, which maps onto a per-client invoice.
StageTypical toolsWhere it runsWhat it hands on
TriggerWatcher script, LIMS, Python APIAnywhereRun folder + samplesheet
Demux + QCbcl-convert, FastQC, MultiQCSlurm CPU nodesPer-sample FASTQ + QC report
Per-assay analysisnf-core-style pipelines (RNA-seq, variants)Slurm, or cloud for allowed stagesCounts, variants, tracks
DeliveryPackaging + QC summarySlurm or head nodePer-client package with lineage
Cost recoveryHorus run costs and budgetsPlatformSpend per user, org, workflow

An illustrative example: one 96-sample run

To make the fan-out concrete, here is an illustrative example (not a measured benchmark): a core runs a 96-sample RNA-seq plate for three clients, 32 samples each.

  • Demux is one task on the whole run folder and produces 96 FASTQ pairs.
  • QC fans out to 96 FastQC tasks, then one MultiQC per client.
  • Analysis fans out to 96 trimming tasks, 96 alignment tasks and 96 quantification tasks. Each is a Slurm job sized to what that tool uses.
  • Delivery is three tasks, one package per client.

That is roughly 400 Slurm jobs that nobody wrote by hand, with three client packages at the end and a cost record for each client's workflow. If a client sends 32 more samples next month, the same pinned workflow release processes them the same way.

How does Horus scale to hundreds or thousands of samples?

Scale in NGS is fan-out: the same analysis over many samples at once. Horus maps a stage over every sample and submits one Slurm job per item, so a 400-sample run is 400 independent jobs that the scheduler packs onto your nodes. Nobody writes a job array or a loop of sbatch calls.

Capacity is handled explicitly. Each machine runs as many items as its capacity allows, and extra items wait for a free slot, so nodes are filled but never oversubscribed. Each stage runs where you allow it: your cluster, your own servers, or cloud machines for the stages you allow. Cloud machines are created on demand when a run reaches such a stage and destroyed after the run.

Per-sample efficiency matters too. On nf-core/rnaseq, Horus with right-sized tasks ran 35% faster and 32% cheaper than Nextflow with nf-core defaults (full nf-core/rnaseq benchmark). The gain came from measuring what every task used and a scripted sizing step that fit tasks to it, which let more samples share a node. Nextflow with an equally tuned config could close that gap; the point is that Horus hands you the measurements.

For sequencing cores and CROs

Sequencing cores and contract research organizations sell turnaround time and reliability. Their pipeline problem is many clients, many assays and a fixed staff. Automation pays off in three places.

Turnaround. The pipeline starts when the run finishes, not when someone reads their email the next morning. Overnight runs are analyzed overnight.

Consistency per client. Each client's analysis runs from a pinned workflow release, so a client who sends samples in March and again in September gets results from the same pipeline version unless you choose to upgrade. Horus campaigns run a workflow release pinned to its project.

Cost recovery. Cores live and die by recharge rates. Horus records compute spend for every run, and budgets per client organization or workflow (cloud spend) stop a client's reanalysis from silently running past the quote. A run whose projected cloud cost would cross the budget is refused before it starts.

Your staff bioinformatician stops being an operator and goes back to building assays. Clients who want to rerun with different parameters can be given runner access to launch the validated workflow themselves from a visual pipeline builder, with no way to edit the pipeline itself.

For pharma and biotech data teams

Pharma and biotech data teams increasingly buy or license external sequencing datasets, from biobanks, consortia, partners and vendors, and have to bring them into a consistent internal analysis. Their problem is heterogeneity: every source arrives in a different layout, at a different depth, sometimes on a different reference.

The automation pattern is an ingestion workflow per source that normalizes into one internal format, followed by the same validated analysis for everything. Because every artifact carries lineage, a downstream result can always be traced to the exact source files, reference and pipeline version, which is what a regulatory or due-diligence review asks for.

Data residency is the other constraint. Licensed patient-derived data often cannot leave the company's own infrastructure. Horus runs on your Slurm cluster and your own machines over SSH with nothing installed on them, and only stages you explicitly assign to cloud ever leave. Instance-type allowlists and budgets let IT decide exactly what can be provisioned, by whom.

For data vendors and marketplaces

Companies that sell processed datasets have a different product: the processed data itself. Buyers pay for consistency, documentation and trust. A dataset processed by a hand-run script on an unknown date is worth less than one with a reproducible, auditable pipeline behind every file.

Three Horus properties map directly onto that value:

  • Repeatable. The same workflow release processes every batch, so batch 40 is comparable with batch 1.
  • Auditable. Lineage records which inputs, parameters, containers and pipeline version produced each delivered file. That record can ship with the data.
  • Cheap at volume. Margins in data products are compute margins. Measuring each task and sizing it to fit, running on hardware you own, and creating cloud machines only for allowed stages and only for the run keeps cost per sample down as volume grows.

Data vendors also tend to process on a schedule rather than per instrument run. The same trigger pattern applies: a watcher sees a new batch and launches the workflow.

Build it or buy it?

Home-grown glue tends to look the same: a few thousand lines of scripts around a LIMS export, owned by one person, undocumented, and frozen the day they leave. The reference architecture above is not hard to describe. It is hard to keep running across instrument upgrades, pipeline releases and staff turnover.

Horus gives you the parts that are tedious to build and maintain: Slurm submission over SSH, on-demand cloud machines for allowed stages, caching, lineage, a visual view of every run, budgets and roles. Your team keeps the parts that are actually yours: which assays you run and how. For a side-by-side of the platforms that could fill this role, read the best bioinformatics pipeline platforms guide.

FAQ

How do sequencing cores automate NGS data processing?

They chain the steps into one pipeline that starts itself: detect a finished sequencing run, demultiplex and QC it, run the assay-specific analysis per client on the cluster or cloud, deliver results, and record compute cost. Horus by Temple Compute can run each of those stages on your Slurm cluster over SSH, or on cloud machines for the stages you allow.

How do sequencing cores deliver data to clients automatically?

Delivery becomes the last stage of the pipeline. Each client gets its own outputs with lineage back to the run, parameters and pipeline version that produced them, so the delivered package is reproducible and auditable without manual bookkeeping.

Can an HPC cluster handle hundreds of samples without manual job scripts?

Yes. A workflow engine fans out one task per sample and submits them to Slurm for you. Horus runs each machine up to its capacity and queues the rest, so a 96-sample or 400-sample run needs no job array or sbatch loop.

How do CROs recover compute costs per client?

By running each client's work under its own organization or workflow and reading spend from there. Horus records the cost of every run, and budgets per user, organization and workflow stop a client's cloud spend from overrunning what was quoted.

Is the Horus RNA-seq output identical to nf-core/rnaseq?

In our benchmark on 12 human samples, the key outputs of the Horus port of nf-core/rnaseq 3.27.0 (Salmon tables, featureCounts summaries, bigWigs) were byte-identical to the Nextflow run, and STAR and Picard metrics matched.