November 8, 2026 · Temple Compute
Fine-Tune a Private LLM on Your Own GPUs and Data
Every organization sitting on sensitive data has the same question now: can we have our own model, trained on our own records, without handing those records to someone else? The answer is yes, and it no longer needs a research lab. An open-weight model, a GPU or two, and a pipeline that keeps every stage on hardware you control is enough.
How do you train a private AI on your own data?
You train a private AI by fine-tuning an open-weight model on your data, on GPUs you control. Prepare and split the data on your own machines, train adapters with LoRA or QLoRA on your GPU servers or HPC cluster, evaluate on held-out data, and keep the checkpoints and reports where the data lives. Nothing needs to reach a third-party API.
The choice of method decides the hardware. These figures are approximate, derived from the memory arithmetic in the QLoRA paper and the Hugging Face PEFT documentation, and move with sequence length, batch size and optimizer:
| Model size | QLoRA (4-bit base) | LoRA (16-bit base) | Full fine-tune |
|---|---|---|---|
| ~1B parameters | 2 to 4 GB | 4 to 6 GB | 16 to 20 GB |
| ~7B parameters | 6 to 10 GB | 16 to 24 GB | 110 to 160 GB, multi-GPU |
| ~13B parameters | 10 to 16 GB | 28 to 40 GB | 200 GB and up, multi-GPU |
A single 24 GB GPU such as an L4 or RTX 4090 covers QLoRA up to 13B and LoRA at 7B. Full fine-tuning is a multi-GPU, multi-node job, and for most private-data use cases it is not worth the cost.
Fine-tune or train from scratch?
Fine-tuning is the right answer for almost every private AI project. A pretrained model already knows language. Fine-tuning teaches it your vocabulary, your formats and your labels with a few thousand to a few hundred thousand examples, on hardware a single team can own. Training from scratch needs orders of magnitude more data and compute.
Pretraining from scratch still has its place: research on model behaviour, tiny specialised models, or settings where you must account for every token the model ever saw. We walk through that path in train a small LLM from scratch, with a pretrain, supervised fine-tuning and evaluation pipeline. That workflow is still in development, so treat it as a reference. For a production use case, start from an open model and fine-tune.
Why keep fine-tuning on your own infrastructure?
Keeping fine-tuning on your own infrastructure removes the biggest risk in private AI: copying sensitive data to a vendor. Patient records, financial transactions, unpublished assay results and customer support logs often cannot leave the institution under contract, regulation or plain common sense. A hosted fine-tuning API means uploading exactly that data.
Running on your own GPUs also changes the economics. Fine-tuning is bursty: heavy GPU use for hours, then nothing. Hardware you already own, or a cluster allocation you already pay for, makes the marginal cost of an experiment close to zero. And the trained weights are yours. There is no per-token inference fee to a provider and no risk that a model you depend on is retired. You decide where it runs and who can query it.
How Horus keeps the data where it is
Horus, by Temple Compute, runs each pipeline stage on the machine you assign it, over plain SSH. Nothing is installed on the target: no agent, no daemon. A GPU server in your rack, a Slurm partition at your institution and a workstation under a desk are all just targets.
That makes the data path explicit. A stage that reads raw records is pinned to a machine inside your network and never runs anywhere else. The training stage goes to your GPU node. Only the outputs you choose to share, such as metrics and a report, need to leave. If you allow a stage to run in the cloud, that is a decision written into the workflow, made in advance, not something that happens behind your back.
For the strictest environments, the open-source Horus engine runs entirely on your own hardware, and private deployment of the full platform is available on the Enterprise plan. Then the orchestration lives inside your network too.
The pipeline: prepare, train, evaluate, report
A private fine-tuning pipeline has four stages, and each one wants different hardware:
- Prepare. Clean the data, map labels, and split train, validation and test with a fixed seed. CPU only, and it should be cached so that changing a learning rate does not redo it.
- Train. Fine-tune the model or adapters on a GPU target. This is the only expensive stage.
- Evaluate. Score the held-out test set and produce metrics and a confusion matrix. Back on CPU.
- Report. Record metrics, training time, hardware and hyperparameters in one document per run, so runs can be compared side by side.
Horus runs these as one workflow with per-stage targets. Each artifact keeps its lineage: which data split, which parameters and which run produced a checkpoint. When an auditor asks how a model was trained, the answer is a link, not an archaeology project.
A runnable example: fine-tuning on transaction data
The fastest way to see this end to end is the
Bank Transaction Categoriser workflow
in Pantheon, our open workflow library. It fine-tunes
distilbert-base-multilingual-cased with Hugging Face Transformers to sort
card transaction descriptions into spending categories.
It has exactly the four stages above: prepare_data, train, evaluate
and report. Training runs on its own target, either CPU or an L4 GPU, and
the script uses CUDA whenever a GPU is present. Each run ends in an HTML
report with metrics, a confusion matrix, training time, hardware and
hyperparameters.
The example data is a synthetic, MIT-licensed sample. Swap in your own CSV with a description column and a category column, point the training target at your GPU machine, and the same workflow fine-tunes on your data without the data going anywhere else. The same shape scales up: replace the encoder with an open LLM and the trainer with a LoRA or QLoRA script.
Sizing GPU jobs without guessing
GPU jobs are where oversized requests cost the most. A training job that asks for eight CPU cores next to a GPU, and uses one, holds seven idle cores for the whole run. Horus measures every task as it runs, including GPU memory and utilization, so the second run can be sized to what the first one used.
That measurement is what produced our headline benchmark. On nf-core/rnaseq with 12 human samples on an AWS Slurm cluster, a right-sized Horus run finished 35% faster and 32% cheaper than Nextflow with nf-core's default resources. The gain came from per-task measurement plus a scripted sizing step. The hands-on method is in sizing GPU jobs.
Keeping GPU spend under control
Private fine-tuning invites experimentation, and experimentation invites runaway GPU bills when cloud capacity is involved. Horus lets you cap GPU spend per team: cloud budgets per user, organization and workflow, a cost check before a run launches, and an allowlist of the instance types anyone may provision.
Cloud GPUs are created only when a run reaches a stage allowed to use them, and destroyed when the run ends. Stages pinned to your GPUs never leave them.
Horus vs a serverless GPU platform
Serverless GPU platforms are excellent when your data can live in their cloud and your team writes Python. They do not run on your own servers or your institution's Slurm cluster, which is exactly where private data usually has to stay. If that is your constraint, see our Modal alternative for your own GPUs.
To try the pattern on your own hardware, start with the Bank Transaction Categoriser, or explore the LLM training workflow for the from-scratch route.
FAQ
Can I fine-tune an LLM without sending my data to the cloud?
Yes. Open-weight models can be fine-tuned entirely on your own GPU servers or HPC cluster. With Horus, each pipeline stage runs on hardware you choose over SSH, so training data and checkpoints can stay on machines you control. For the strictest setups, the open-source engine or a private deployment keeps the orchestration inside your network too.
Should I fine-tune an existing model or train an LLM from scratch?
Fine-tune, in almost every case. Fine-tuning adapts a pretrained model to your data with a fraction of the compute and data that pretraining needs. Training from scratch makes sense for research, very small specialised models, or when you must control every token the model has seen.
How much GPU memory do I need to fine-tune a 7B model?
Approximately 6 to 10 GB with QLoRA, 16 to 24 GB with LoRA on 16-bit weights, and well over 100 GB across several GPUs for a full fine-tune. Sequence length and batch size move these numbers, so measure your own run.
What is the difference between LoRA and QLoRA?
LoRA freezes the base model and trains small low-rank adapter matrices, which cuts memory and checkpoint size. QLoRA does the same on top of a base model quantized to 4 bits, which cuts memory further so larger models fit on a single smaller GPU.
Can Horus run fine-tuning jobs on a Slurm cluster?
Yes. Horus submits each stage as a Slurm job over SSH with your own account, with no agent installed on the cluster. Data preparation can run on CPU nodes and training on a GPU partition in the same workflow.