This basically enables KV cache reuse when there is a prefix matching (from my shallow understanding of how KV cache works).
I failed to see how this help for local deployed LLM, unless you consider the case you ask the same question or with the same prefix are high (like always starts with "please help me ..."?)","title":null,"type":"comment","url":null},{"author":"fredliu","children":[],"created_at":"2023-09-14T18:46:52.000Z","created_at_i":1694717212,"id":37513072,"options":[],"parent_id":37510548,"points":null,"story_id":37509659,"text":"I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?","title":null,"type":"comment","url":null}],"created_at":"2023-09-14T15:47:58.000Z","created_at_i":1694706478,"id":37510548,"options":[],"parent_id":37510415,"points":null,"story_id":37509659,"text":"Just from the abstract, this is primarily for batching inference, for batched inference using GPUs gives an order of magnitude speed increase so probably not something that usually makes sense to do on CPUs\u2026","title":null,"type":"comment","url":null},{"author":"brucethemoose2","children":[],"created_at":"2023-09-14T16:30:18.000Z","created_at_i":1694709018,"id":37511122,"options":[],"parent_id":37510415,"points":null,"story_id":37509659,"text":"Only if the CPU is serving multiple users, maybe.
LLMs can't batch token generation for single users. Its sequential, each token depends on the next. In fact that's a part of the paper: "dumb" batching will leave the GPU underutilized because responses aren't all the same length, and they end up processing one token at a time at the end.","title":null,"type":"comment","url":null}],"created_at":"2023-09-14T15:37:56.000Z","created_at_i":1694705876,"id":37510415,"options":[],"parent_id":37509659,"points":null,"story_id":37509659,"text":"Without understanding most of that paper, here's a question for someone who might know more: can pagedattention work to make cpu inference faster too?","title":null,"type":"comment","url":null},{"author":"notpublic","children":[],"created_at":"2023-09-14T17:28:43.000Z","created_at_i":1694712523,"id":37512023,"options":[],"parent_id":37509659,"points":null,"story_id":37509659,"text":"source code: https://github.com/vllm-project/vllm","title":null,"type":"comment","url":null},{"author":"heliophobicdude","children":[{"author":"yelite","children":[{"author":"heliophobicdude","children":[],"created_at":"2023-09-22T20:28:25.000Z","created_at_i":1695414505,"id":37617335,"options":[],"parent_id":37515229,"points":null,"story_id":37509659,"text":"Thank you so much for the response! This is really good info!","title":null,"type":"comment","url":null}],"created_at":"2023-09-14T21:41:36.000Z","created_at_i":1694727696,"id":37515229,"options":[],"parent_id":37513383,"points":null,"story_id":37509659,"text":"> How doesn't paging worsen speed performance though?
It does worsen the performance of the attention kernel, if comparing to kernels which takes keys and values in continuous memory layout.
> Wouldn't the speed improvements be coming from that instead? Don't put an expected short input and output in the same batch as a big input and big output?
Actually it puts everything in the same batch. The reason for its high throughput is that sequences are removed from the batch as soon as it's finished, and new sequences can be added to the batch on-the-fly if there is enough space in KV cache. This is called continuous batching (https://www.anyscale.com/blog/continuous-batching-llm-infere...).
Paged attention and "virtualized" KV cache play an important role in an efficient implementation of continuous batching. Text generation in LLM is a dynamic process and it's not possible to predict how long the output is when scheduling incoming requests. Therefore a dynamic approach is needed for KV cache allocation, even though it hurts the performance of attention.","title":null,"type":"comment","url":null}],"created_at":"2023-09-14T19:09:23.000Z","created_at_i":1694718563,"id":37513383,"options":[],"parent_id":37509659,"points":null,"story_id":37509659,"text":"Ah, I see. This isn't necessarily virtualizing the static weights but the variable -sized and data dependent key value caches. These caches are built up as you go through the sequence of tokens. Makes sense.
How doesn't paging worsen speed performance though? If you are making more trips to the memory, then are you really just saving vram?
Also I see that vLLM which implements PagedAttention is also using a better scheduling? Wouldn't the speed improvements be coming from that instead? Don't put an expected short input and output in the same batch as a big input and big output?
What are the results of using the sequence-length only without virtualization?","title":null,"type":"comment","url":null}],"created_at":"2023-09-14T14:42:03.000Z","created_at_i":1694702523,"id":37509659,"options":[],"parent_id":null,"points":102,"story_id":37509659,"text":null,"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","type":"story","url":"https://arxiv.org/abs/2309.06180"}