Conventional MD with AMBER: Replicas on Slurm, 76% Fewer CPU Cores

Conventional MD is the workhorse of structure-based work. Build a solvated system, equilibrate it, run production, analyze. Every group has a script for it, and every script carries #SBATCH lines that someone copied years ago.

This post walks through the conventional MD workflow in our open workflow library, a port of a lab's AMBER protocol into a Horus workflow. It also covers what happened when the group running that protocol could finally see what their jobs used: they cut their CPU core requests by 76%.

What the workflow does

The workflow goes from a PDB ID, or an uploaded structure, to a clustered HTML report. It has 10 tasks:

fetch_pdb            PDB ID or upload          --> structure.pdb
extract_ligand       structure + residue name  --> ligand pose
parameterize_ligand  pose                      --> GAFF2 mol2 + frcmod (AM1-BCC)
build_complex        structure + ligand        --> prmtop, inpcrd, build report
build_system         protein-only variant
prepare_replicas     renders .mdin templates   --> one slot per replica
equilibrate (xN)     10-step equilibration, one GPU job per replica
produce     (xN)     chunked production, one GPU job per replica
analyze     (xN)     cpptraj: RMSD, RMSF, Rg, H-bonds, ligand contacts
report               k-means over all replicas --> report.html, summary.csv

The default input is 3HTB, T4 lysozyme L99A with 2-propylphenol bound. The build step protonates the protein with PROPKA at pH 7, uses ff14SB and TIP3P, puts it in a truncated octahedron with 12 Å padding, and adds 150 mM NaCl on top of neutralization. Every choice and warning goes into prep_report.json.

Each replica then runs a 10-step equilibration: minimization, heating, restrained NPT with the restraints stepped down, a second minimization, then free NPT and NVT. That is 15 ns at the default scale. Production runs 200 ns per replica in 10 ns chunks. The engine is OpenMM from a conda environment, which runs on any NVIDIA GPU node, or pmemd.cuda if your site has licensed AMBER. Both run the same rendered .mdin files.

The protocol itself is ported from MolBioMedUAB/protocols, the create_md_custom.sh script. The physics did not change. The plumbing did.

From one big job to one job per replica and phase

The original script ran as a single Slurm job with one GPU. The workflow splits it up. prepare_replicas writes one self-contained folder per replica. Then a map fans out over those folders, so each replica gets its own equilibration job, and a second map gives each replica its own production job.

Splitting the work this way gives you three things:

• Separate walltimes. Equilibration asks for 5 hours, production for 10 days. A short job with a short limit tends to start sooner than one long job that has to reserve the whole budget.
• Replicas in parallel. With three replicas you get three GPU jobs at once instead of three runs back to back.
• Restarts that skip finished work. If a production job dies on day six, the re-run does not repeat that replica's equilibration, or any replica that already finished.

Analysis runs next to the trajectories, on the cluster, so the large .nc files never travel. Only the per-replica analyses come back for clustering and the report.

Watching a run that takes ten days

A 200 ns production run is a long time to wait for an mdout you cannot see. The workflow ships a small live monitor that runs beside the MD engine. About every 30 seconds, Horus pulls its digest off the cluster into the task's side products:

• the newest mdout and mdinfo, with ns/day and the ETA
• every energy record so far, as CSV and a plot of EPtot, temperature and density
• the newest trajectory chunk, stripped of solvent, ready to open in Mol*
• the CA RMSD of those frames

So a blown-up density or a ligand drifting out of the pocket shows up in the first hours, not at the end of the allocation.

Where the 76% came from

The group running this protocol had the usual problem with HPC requests. They did not know what their jobs used, so they asked for more than they needed.

The information existed, but it was hard to get at. Slurm records usage after a job ends, one job at a time, through sacct or seff. Nobody does that for every replica of every run. The #SBATCH header gets padded once and then copied forward.

GPU MD is where this hurts most. With pmemd.cuda or OpenMM's CUDA platform, the simulation runs on the GPU. The CPU cores mostly drive the GPU and write output. Extra cores requested next to the GPU sit idle for the whole run, and in this workflow that run is ten days long.

Horus measures every task as part of the run. Its resource monitor plugin wraps each command and records:

• %CPU, peak memory and wall time from /usr/bin/time -v
• GPU memory and utilization from nvidia-smi
• a one-second time series of the task's processes, uploaded as a side product

It reads these back over the same connection Horus uses to run the job, so it works the same on a Slurm cluster as on a laptop. At the end of the run, every task is in one summary table and one JSON report, across all replicas.

With that table in front of them, the group set the CPU cores for each job to what the job used. Their CPU core requests dropped by 76%. The simulations did not change. The same replicas ran with the same protocol on the same GPUs. They were just no longer holding about three cores in four that did nothing.

That matters beyond tidiness. On most Slurm sites, allocated cores count against your fair-share usage whether you use them or not. Smaller requests also tend to fit into scheduling gaps sooner. On a cloud or billed cluster, idle cores you reserved still show up on the invoice.

Running it

The workflow lives in Pantheon, our open workflow library. Out of the box every task runs locally with resource hints. On a Slurm cluster, fill in partition and account in the three Slurm blocks at the top of workflow.yaml and point the tasks at them.

Once a first run finishes, read the resource report. Then set cpus_per_task and mem to match what the replicas used, rather than what the old header said.

For a lighter system setup without the full replica pipeline, see AMBER Protein-Ligand Complex MD Setup. To run many independent simulations as one study, the same fan-out pattern is covered in automating parametric studies.

FAQ

What is conventional MD?

Conventional MD (cMD) is plain molecular dynamics without enhanced sampling: no bias potentials, no replica exchange, no accelerated MD. Sampling comes from running long enough, and from running several independent replicas.

Why run several MD replicas instead of one long trajectory?

Independent replicas, each started with different random velocities, sample different regions of conformational space and give you an error estimate. One long trajectory can stay stuck in a single basin. Replicas also run in parallel on separate GPUs.

How many CPU cores does pmemd.cuda need?

Very few. The simulation runs on the GPU, and the CPU mostly drives it and writes output. Measure your own system before setting cpus_per_task. Do not copy a header from a CPU-only job.

How do I find what a Slurm job actually used?

After a job ends, sacct and seff show its CPU efficiency and peak memory, one job at a time. Horus records the same numbers for every task while the run happens and puts them in one report across the whole workflow.

Open Horus to run conventional MD replicas on your own cluster or cloud GPUs.