2 Commits

Author	SHA1	Message	Date
Jesse Gross	b7b2aa5d4e	mlxrunner: Cache.Update takes ForwardBatch and returns KVHistory ... Signature changes from Update(k, v) to Update(batch, k, v) returning (k, v, KVHistory). KVCache returns a real page table mapping positions to buffer slots. RecurrentCache returns empty KVHistory from Update. Replace Cache.Offset() with Offsets() returning per-sequence offsets. Add KVHistory type to mlx package.	2026-04-03 19:50:41 -07:00
Jesse Gross	96e36c0d90	mlxrunner: share KV cache across conversations with common prefixes ... Enable multiple conversations to reuse cached computations when they share token prefixes (e.g. the same system prompt). A prefix trie tracks shared regions so switching between conversations only recomputes tokens that diverge. Inactive conversation state is paged from active GPU memory to other memory and restored on demand, with LRU eviction to keep memory usage bounded.	2026-03-18 16:06:33 -07:00