TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Paper
•
2404.11912
•
Published
•
16
Papers - Speculative Decoding - KV Cache
we recognize two memory bottlenecks: model weights and KV cache, and the latter gradually bottleneck(s) as context length increases