High Performance LLM Inference in Pure Python with PyTorch Custom Ops - Yineng Zhang
PyCon US · 18:26
Yinan Zhang (senior director of inference at Tigon AI) argues that the "Python vs C++" framing for LLM inference is obsolete: in modern serving systems Python is an orchestration layer sitting on top of native kernels...