Why AI Spent 60 Years Hunting for the Right Activation
Jia-Bin Huang · 26:25
The activation inside a Transformer FFN is a tiny per-dimension nonlinearity that decides which signals pass, and sixty years of design have been a hunt for functions that stay differentiable, avoid same-sign or vanis...