Jongryool Kim - Disaggregated LLM Serving with Shared Memory KV Cache at Rack Scale
Berkeley RDI · 4:30
SK Hynix's Jung Wook Kim argues that a disaggregated, physically separate memory pool shared by several servers can speed up LLM serving. In SK Hynix's demo, the pool holds the KV cache so that prefill and decode node...