One GPU cluster, five kinds of work

This round grows. The interviewer gives the opening, then adds a requirement about every 10 minutes. Say what changes in your design each time, and what it costs.

Opening (minute 0). Halvard Autonomy (fictional) builds self-driving trucks. It owns one cluster: 64 nodes with 8 H100 GPUs each, 512 GPUs in all. The nodes sit in 4 pods of 16. Inside a pod the fabric is non-blocking. Between pods the spine is 2:1 oversubscribed. Today an HPC team runs Slurm on it, and a platform team runs Kubernetes for services on other hardware. The CTO asks: "Slurm, Kubernetes or both, for this cluster?"

Workloads today:

• Perception training: one 256-GPU (32-node) run each week, 60 hours long, checkpointing every 30 minutes. It must sit inside 2 pods.
• Planning research: about 40 jobs each weekday, 8 to 64 GPUs (24 on average), 4 hours on average.
• Simulation: replays of driving logs, 1-GPU tasks, as many as the cluster can take. They checkpoint every 10 minutes and can be stopped at any time.

Added at minute 10. The remote-assist inference service moves onto this cluster. It needs 24 GPUs at all times and 48 during the daytime peak (10 hours on weekdays), with 99.9% availability. If a node fails, its replicas must be serving again within 2 minutes.

Added at minute 20. 40 engineers want Jupyter notebooks with a GPU from 9 to 6 on weekdays. Measured over a month, they actually use about 8% of a GPU each.

Added at minute 30. The safety-validation team runs tests for the regulator: 64 GPUs for 12 hours, twice a week, booked 48 hours ahead. No other workload may share their nodes during a run, and each run must be reproducible: same driver, same container image.

Added at minute 40. From next quarter, the perception team wants a bigger weekly run: 384 GPUs (48 nodes) for 40 hours, in place of 256 GPUs for 60 hours. It still checkpoints every 30 minutes.

Added at minute 50. Finance wants a monthly GPU bill per team, and a target of 75% of GPU-hours in use.

Asks, at every stage:

1. What does the scheduler need to do now that it did not before?
2. Which mechanism does it (in Slurm, in Kubernetes, or both)?
3. What does it cost in GPU-hours or risk?

Write down, for each workload in the opening, what it needs from a scheduler (gang start, topology, preemption, fair share). Then pick Slurm, Kubernetes or both, in one sentence. You will revise it as requirements arrive.