TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Paper
•
2404.11912
•
Published
•
16
we utilize a 4K retrieval cache as an intermediate draft cache in our hierarchical system, while leveraging the JackFram/Llama68M (JF68M) [28] model