Does not work with text-generation-webui.
I am using oobabooga in CPU only mode and always get the following error. I have tried all models with different quants. I have also reinstalled oobabooga.
Other gguf models work just fine with llama.cpp in oobabooga. But I still get the following error:
File "E:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\llama_cpp\llama_cpp.py", line 498, in llama_load_model_from_file
return _lib.llama_load_model_from_file(path_model, params)
OSError: exception: access violation reading 0x0000000000000000It seems like this is a problem with oobabooga rather than the model files as they seem to work with llama.cpp.
Can you confirm this? What's the solution? Just waiting for text-generation-webui to catch up?
try going into the command line for oobabooga (cmd_windows.bat or cmd_linux.sh or whatever you use), and upgrade llama-cpp-python manually. (pip install --upgrade llama-cpp-python) At least that's what I had to do, and also I had to check the 'cpu' flag. Lastly, I found that the StarChat preset work better than Divine Intellect for Deepseek-coder.
In my case, i have it running as a service in a conda environment. Just stop service, start environment, do the upgrade piece, then restart service. I have not tried starchat, i have been using vicuna 1.1 with success in general, including this model.. ( tho i think that ajibawa-2023/Python-Code-13B/33B ( gguf format for me ) is doing much better for output for me, at least for python. But not done a lot of testing yet , just some early testing.