Beyond Softmax — Linear Attention
Jia-Bin Huang · 34:31
Linear attention removes the softmax from attention. This turns the ever-growing KV cache into a fixed-size matrix "hidden state," so memory stays constant no matter how long the sequence gets. The video explains the...