Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operations
Jane Street · 63:21
Horace He (Meta PyTorch compilers; torch.compile, Flex Attention) argues that frontier training is ~1e26 FLOPs of simple matmuls on unprecedented hardware, and the hard systems problem is not inventing more compiler t...