I Made My AI Agents Faster Without a GPU Upgrade
Alex Ziskind · 15:46
KV caching speeds up local coding agents a lot. Without a cache, every turn makes the model reread the whole conversation, so a busy GPU gets slower and slower. vLLM's built-in prefix cache, which is on by default, pl...