12. Running on HPC

The 1D chain examples of this tutorial run on a laptop, but real research systems (devices with hundreds of atoms, many biases, length series) require a cluster. This chapter organizes the parallel structure of SIESTA/TranSIESTA/TBtrans calculations, job-script writing, and the habit of verifying immediately after submission into a workflow. Partitions, modules, and core counts differ from system to system, so all of them are written as placeholders — substitute the documentation of your own cluster.

Learning objectives

• Identify the cost structure of each workflow stage and its parallelization axis.
• Write SLURM/PBS job scripts and fan out independent calculations.
• Make a habit of measured verification 1–2 minutes after submission and of detecting early termination.
• Reuse DM/TSDE restart assets to reduce the cost of SCF iterations.

Where the cost is

Stage	Cost	Parallelization axis	Notes
Electrode SCF (Ch. 5)	Low	k-points, diagonalization	Computed once and shared by all devices
Device 0 V TranSIESTA (Ch. 6)	High	Diagonalization/inversion, k (if any)	A Green function inversion at every contour point
Finite-bias TranSIESTA (Ch. 9)	High × number of biases	Same as above	The bias chain is sequentially dependent
TBtrans (Ch. 7)	Medium	Energy points	The best parallel scalability
sisl post-processing (Ch. 10)	Low	—	Local machine is sufficient

TranSIESTA, dominated by matrix operations, grows as $O(N^3)$ in the number of orbitals $N$. TBtrans, by contrast, has fully independent energy-grid points — even our example has 6001 points over $[-3, 3]$ eV with delta 0.001 eV — so distributing points over MPI ranks scales almost ideally.

MPI parallelism in SIESTA/TranSIESTA

• Diagonalization parallelism — the SCF diagonalization of SIESTA is distributed with the ScaLAPACK family, and depending on the build a high-performance solver such as ELPA can be used (Diag.Algorithm — check the build/manual for availability). Increasing the rank count splits the matrix into finer blocks and raises communication cost, so pouring ranks onto a small system is actually slower. Distributed diagonalization libraries are sensitive to the processor-grid shape, and the numerical behavior can change with the rank count — keeping the rank count fixed within a project is good for reproducibility.
• k-point parallelism — with several k-points there is almost no communication between them, so the efficiency is good. However, the device stage of a 1D transport calculation has few k-points (transverse is vacuum, the transport direction is treated as semi-infinite), so the gain along this axis is limited. The electrode SCF has dense k along the transport direction, so the k-parallel gain there is large.
• Rule of thumb — first run a small scaling test on one representative production-size case, doubling the rank count each time, and pick the point that optimizes throughput per unit resource (e.g. core-hours per SCF step) rather than wall-clock time.

Fanning out independent jobs

Draw the dependency graph first:

• Sequential dependence: electrode → device 0 V → bias chain (each bias restarts from the converged result of the previous one, Ch. 9).
• Mutually independent: the $+V$ chain and the $-V$ chain (both branch from 0 V), device length series, parameter scans, different structural cases.

For $N$ independent calculations, submitting them in parallel as $N$ jobs (or a job array) is overwhelmingly better in wall-clock time than running them sequentially on one large node. Running a single case in one large allocation is the archetype of wasted resources.

Job scripts

SLURM example

run.slurm

#!/bin/bash
#SBATCH --job-name=c19n-0bias
#SBATCH --partition=<partition>          # replace with your system's partition name
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=<ncores>       # match the number of physical cores per node
#SBATCH --time=06:00:00
#SBATCH --mail-type=BEGIN,END,FAIL       # mail notification on start/end/failure
#SBATCH --mail-user=<your-email>

module load <siesta-module>              # replace with your system's module name

srun siesta input.fdf > siesta.stdout && touch NORMAL_EXIT

PBS example

run.pbs

#!/bin/bash
#PBS -N c19n-0bias
#PBS -q <queue>
#PBS -l select=1:ncpus=<ncores>:mpiprocs=<ncores>
#PBS -l walltime=06:00:00
#PBS -M <your-email>
#PBS -m abe

cd $PBS_O_WORKDIR
module load <siesta-module>

mpirun -np <ntasks> siesta input.fdf > siesta.stdout && touch NORMAL_EXIT

Include the mail-notification options (--mail-type / -m abe) as a habit — they keep you from discovering the failure of a long job hours later. The NORMAL_EXIT marker on the last line is a convention for mechanically filtering "jobs that terminated normally" in batch pipelines. But the presence of the marker does not by itself mean convergence — the judgment is always made from the convergence/normal-termination messages at the end of the stdout.

Verification right after submission — do not stop at "it's submitted"

The most common waste of time on HPC is leaving a job unattended after submission and discovering the next day that it failed right after starting. Fix the following three steps as a checklist.

Time	Check	Example command
1–2 min after submission	Whether the queue state is R (if PD, the reason for waiting), whether the stdout file appeared, whether there is an error causing an immediate abort	squeue -u $USER or qstat -u $USER, ls -l siesta.stdout, the grep below
5–10 min	Whether the SCF iterations are actually progressing	Whether the value of grep -c scf siesta.stdout is increasing
30–60 min	First case complete or sufficient progress. If stalled, diagnose the cause	stdout mtime + step count

The grep for immediate-abort errors is a one-liner:

grep -iE "error|abort|segfault|out of memory" siesta.stdout

Additionally, treat two signals as flags:

• Early termination — a job that finished in a fraction of the expected walltime is more likely a failure signal than a success (input error, immediate abort due to a missing file). Do not treat it as complete before verifying the output.
• A hung job — even if the queue state is R, if the stdout has not been updated for a long time (check the last modification time) the job may in fact be stalled. The triple check of queue state + stdout update + step increase is the reliable one.

For fan-out jobs, compare the number of output files per case against the number of nodes/jobs — this catches, within the first hour, the case where only some ran and the rest quietly dropped out.

Restarting: do not throw away converged assets

The cost spent on SCF convergence survives as files. Reusing these assets on recalculation, extension, or migration is the default.

• SIESTA density matrix — with DM.UseSaveDM T enabled, the SCF warm-starts from the *.DM in the same directory. Even a recalculation with a slightly changed structure converges much faster than a fresh one.
• TranSIESTA TSDE — in a finite-bias chain each bias starts from a copy of the previous bias's *.TSDE (DM+EDM) (Ch. 9). Without it, TranSIESTA stops immediately with an error. The distinction between files that may be copied and files that must never be copied (bias-dependent caches) is organized in the pitfall collection.
• Preparing for walltime overrun — assume that long relax/SCF runs may not finish within the walltime: always enable restartable options and split the submissions.

Exercises

1. Run the device 0 V calculation with the rank count doubled each time, measure the wall-clock time per SCF step, and find the rank count that minimizes "core-hours per step". Is it the same as the wall-clock minimum?
2. Draw the dependency graph of the four bias calculations $\pm 0.1$ and $\pm 0.2$ V, and design the submission strategy (how many jobs, in what order) that finishes in minimum wall-clock time.

References

• SIESTA/TBtrans manual: https://docs.siesta-project.org/
• The user guide of your own cluster (partitions, modules, scheduler policies)