Distributed Training by Nous Research @ Nouscon 2024
Nous Research · 14:52
Nous Research chief scientist Bowen Peng argues that making individual chips bigger is running into physical limits, so future AI scaling depends on distributed training across many GPUs. The obstacle is communication...