An AI forecasting platform for a weather agency

Your brief, sent 5 days before the interview. Prepare a 30-minute presentation of the AI infrastructure you would propose, with 7 to 12 slides. A panel will interrupt with questions. Then 30 minutes of open questions.

Calder Meteorological Agency (fictional) is a national weather service.

Business facts:

1. It runs physics-based forecasts on a CPU supercomputer managed with Slurm, 4 forecast cycles a day.
2. It wants an AI forecast model in daily operations for the next storm season: live in 15 months.
3. Budget: €14 million of capital over 2 years, including 3 years of support. Everything stays on premises in the country, because this is an emergency service.
4. Team: 6 HPC engineers who know Slurm, a parallel file system and InfiniBand from the CPU system, and 4 ML researchers. Nobody has run Kubernetes.
5. Forecasts feed the civil-protection alert service. A late forecast cycle counts as a public-safety incident.

Technical numbers from the agency:

1. One full training run of their model took 110,000 H100 GPU-hours on a rented prototype. They plan 4 retrains a year, plus experiments worth 2 more full runs: 6 runs a year. A full run must finish within 4 weeks.
2. Training data: 210 TB after preprocessing. A run reads it 12 times, in random order.
3. Each forecast cycle runs 50 ensemble members. A member is 40 steps of 6 hours (10 days ahead), and each step takes 1.1 s on one H100. The whole ensemble must finish within 20 minutes of the input data arriving.
4. Output: 2.6 GB per member per step. Keep 30 days of output online.
5. Facility: 14 free rack positions, air-cooled, up to 25 kW per rack. Power is not a limit.

Assume:

• An 8-GPU H100 node costs €300,000 with 3 years of support and draws up to 10.2 kW.
• The InfiniBand fabric costs €35,000 per node. Parallel storage costs €800 per usable TB.
• Two of the 14 rack positions go to network and storage.
• Electricity costs €0.18 per kWh, with a PUE of 1.35. Nodes average 70% of their peak draw.

Before you make any slide, write the one sentence the agency's director should remember, and the five numbers that go on your sizing slide.