Fritsche Lab · University of Michigan

SLURM Playbook

Start with one working task. Scale to independent jobs, measure what they use, and recover only what failed.

Start here Explore the example

Command lookup

From one task to a workflow

Independent tasks. One reproducible result. Combine only after all tasks succeed.

MeasurablePilot, inspect, and right-size requests.

ModularOne stable work unit per task.

RecoverableKeep good results. Rerun failures.

This is the lab reference to reach for when you have a concrete question: How many CPUs should I request? Why is my job pending? How do I rerun only task 11? Where should the combine step run?

If you can already run a command in a Unix shell, the guide will help you move it into a single-node CPU job or array on U-M’s SLURM clusters. It covers serial and threaded programs, single-node process pools, independent work units, one fan-in stage, accounting, and recovery. For MPI, multi-node jobs, GPUs, cluster administration, or project-specific compliance decisions, start with ARC or the appropriate specialist instead.

Treat each resource request as a hypothesis: pilot it, inspect the evidence, and revise it.

Start with the problem in front of you

CPU or array?Decide whether the program is serial, threaded, multi-process, or many independent repetitions. Right-size a requestUse representative pilots, seff, sacct, MaxRSS, elapsed distributions, and thread settings. Parallelize repeated workPut one stable work unit per manifest row and map rows to array task IDs. Chain pipeline stagesFan out with an array, then fan in only after every upstream element succeeds. Diagnose a failureFollow the evidence in order: state/reason → logs → exit code → resources → environment → paths/I/O. Inspect the working exampleSee how the manifest, worker, safety guard, submission wrappers, and result files fit together.

The job lifecycle

stateDiagram-v2
    accTitle: SLURM job lifecycle
    accDescr: A job moves from script creation through submission, pending, running, completion, measurement, and revision.
    state "Write or revise script" as Script
    state "Pending: squeue / scontrol" as Pending
    state "Running: logs / sstat" as Running
    state Outcome <<choice>>
    state "Completed: measure with seff / sacct" as Measure
    state "Diagnose logs and exit code" as Diagnose
    state "Revise request or code" as Revise
    [*] --> Script
    Script --> Pending: sbatch
    Pending --> Running: resources available
    Running --> Outcome: terminal state
    Outcome --> Measure: completed
    Outcome --> Diagnose: failed, timed out, or cancelled
    Measure --> Revise
    Diagnose --> Revise: correct root cause
    Revise --> Script

The scheduler is not the debugger. SLURM tells you where to look through state and reason; the application log and exit code tell you what the process actually did.

Six operating principles

1Prove one task locally.
Prefer a small or synthetic smoke test.

2Request only CPUs the code can use.
Allocating CPUs does not make serial code parallel.

3Tune from distributions.
Use seff and array-wide sacct, not one easy task.

4Represent work in a stable manifest.
One auditable row should always mean the same task.

5Throttle arrays and use unique logs.
Protect account limits and shared storage.

6Aggregate after success; rerun only failures.
Preserve completed work and recover at task granularity.

A working first path

Open Start Here and run one synthetic manifest row locally.
Run the complete scheduler-free test and confirm the documented PASS line.
Validate the submission command with --dry-run, then perform the small Great Lakes acceptance run when you have an account and partition.

The worked example requires Bash 3.2+ and Python 3.10+. A SLURM account is not needed for the local checkpoints.

The example is tested on Great Lakes. You can use the same scheduler pattern on Armis2 after checking its current account, partition, storage, billing, and sensitive-data rules in the U-M Cluster Reference; a duplicate training run is not needed.

Optional downloads

The ZIP expands to a directory named slurm-example-pipeline/. The PDFs are teaching aids; the web pages and tested example are the versions we keep current.

Improve this resource

If a command is unclear, an example fails, or a U-M policy appears outdated, open an issue. Include the page and non-sensitive evidence, and link an official policy source when reporting policy drift.