anthracite-org

/

magnum-v2-12b

like 69

We can't provide support for third party engines, but I'd try updating to v0.3.6 in hopes they merged / fixed nemo inferencing, if that still doesn't work I'd open an issue on their github: https://github.com/ollama/ollama/issues hope that helps

Do you have any recommended engines that can run on Ubuntu?
The models your team has developed are really impressive.

tabbyAPI works great with it (exl2), but you could try koboldcpp too (gguf) or if you want to load fp16 you can give vllm or aphrodite (a little more experimental) a go. (though afaik tabby can load fp16 too)

personally I only run it in tabby so I can not vouch for any of the other options.

I'll try loading fp16 with tabby, as I've been wanting to experience it. I’ve tried other quantized models and they’ve been great. Thank you for your efforts and sharing!