Doesn't load on Runpod
I'm using the template for "TheBloke Local LLMs One-Click UI and API" template using an L40. When I attempt to load the model, I get the following error:
Traceback (most recent call last):
File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 213, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
File "/workspace/text-generation-webui/modules/models.py", line 87, in load_model
output = load_func_maploader
File "/workspace/text-generation-webui/modules/models.py", line 311, in AutoAWQ_loader
model = AutoAWQForCausalLM.from_quantized(
File "/usr/local/lib/python3.10/dist-packages/awq/models/auto.py", line 50, in from_quantized
model_type = check_and_get_model_type(quant_path, trust_remote_code)
File "/usr/local/lib/python3.10/dist-packages/awq/models/auto.py", line 25, in check_and_get_model_type
raise TypeError(f"{config.model_type} isn't supported yet.")
TypeError: mixtral isn't supported yet.