Ship an 840 GB checkpoint to 1,536 hosts

You run the serving fleet for Brightwater AI (fictional). A new version of its largest model ships every week. Each serving host holds one full copy of the model.

The checkpoint:

• 840 GB in FP8 (a mixture-of-experts model), stored as 168 files of 5 GB in a regional object store, with a signed manifest that lists each file's SHA-256 hash.

The fleet (assumptions):

• 1,536 serving hosts in one region, in 48 racks of 32.
• Each host has 8 GPUs with 141 GB of memory each (1,128 GB in all), 2 TB of RAM, one 100 Gb/s front-end NIC (12.5 GB/s) for storage and peer traffic, and local NVMe that writes at 8 GB/s and reads at 12 GB/s.
• Host-to-GPU copies run at about 200 GB/s per host.
• Each rack switch has 800 Gb/s (100 GB/s) of uplink.
• The object store gives this account 200 Gb/s (25 GB/s) of egress in total.
• The GPU fabric (InfiniBand) is reserved for model traffic and is off limits for now.

Rules:

• Serving capacity must stay at or above 85% during the switch.
• No host serves the new version until it has proved that it holds the right weights.

Asks:

1. What is the best possible time to get the new weights onto every host? Show the bound and what sets it.
2. Compare three plans: every host downloads from the object store; a tree that forwards whole files; your design. Give a time for each.
3. Design the transfer: chunks, who sends to whom, integrity checks, racks.
4. How does a host prove it is ready, and how do you switch 1,536 hosts while keeping 85% of capacity?

Write the lowest possible time to get the new weights onto one host, and what sets it. Then write the time for the naive plan where every host downloads from storage.