How to Train Models Too Big for One GPU | ZeRO & FSDP
Jia-Bin Huang · 23:05
Training a model too big for one GPU takes two techniques. Mixed precision runs the computation in 16-bit and keeps a 32-bit copy for the optimizer, which comes to 16 bytes per parameter. ZeRO (Zero Redundancy Optimiz...