How do you use / load these models?
#1
by
Arya123456
- opened
Hey. This sounds like a very interesting project but I can't load these models in Oobabooga WebUI and the app crashes. All other EXL2 or GGUF models work fine though.
Can you please share how you use these models? Thank you :)
Thank you for your interest in our project. Our project is open-sourced at https://github.com/microsoft/VPTQ, and you can quickly try it out on a GPU. Additionally, we are working on supporting llama.cpp, so please stay tuned!