Models fails to load in ooba, exllamav2 and exllamav2_hf loaders
#1
by
nsfwthrowitaway69
- opened
This is occurring also with the 5.0 bpw quant. Haven't tried other quants yet, so can't say for certain whether they have issues as well. I've tried multiple different GPUS (renting on runpod.io) and am able to lod other exl2 quants of models, just not any of Aetheria.
I just tested the 2.4 and 5.0 models and they both work. Please make sure that your ooba is up-to-date.
2023-12-11 11:14:34 INFO:Loading LoneStriker_Aetheria-L2-70B-5.0bpw-h6-exl2-2...
2023-12-11 11:14:57 INFO:LOADER: ExLlamav2_HF
2023-12-11 11:14:57 INFO:TRUNCATION LENGTH: 4096
2023-12-11 11:14:57 INFO:INSTRUCTION TEMPLATE: ChatML
2023-12-11 11:14:57 INFO:Loaded the model in 22.11 seconds.
Output generated in 35.31 seconds (16.68 tokens/s, 589 tokens, context 1913, seed 611086752)
2023-12-11 11:17:33 INFO:Loading LoneStriker_Aetheria-L2-70B-5.0bpw-h6-exl2-2...
2023-12-11 11:17:41 INFO:LOADER: ExLlamav2
2023-12-11 11:17:41 INFO:TRUNCATION LENGTH: 4096
2023-12-11 11:17:41 INFO:INSTRUCTION TEMPLATE: ChatML
2023-12-11 11:17:41 INFO:Loaded the model in 7.80 seconds.
Output generated in 23.32 seconds (16.46 tokens/s, 384 tokens, context 1913, seed 20034736)