Regrading bonito-v1_q8_0.gguf
#2
by
KrishGoyani
- opened
how can i use bonito-v1_q8_0.gguf model in bonito library's Bonito function?
I recommend using AWQ, the GGUF version is here mainly for batched inference. You can check how to use the AWQ version here: https://github.com/BatsResearch/bonito/blob/main/tutorials/Quantized_Bonito_Tutorial.ipynb