ISTA-DASLab

/

Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16

like 20

Follow

IST Austria Distributed Algorithms and Systems Lab 46

It would be very nice if there was a Yi 34B aqlm, DeepSeek 33b AQLM already fits on an RTX 3060 12gb and works very well. It is also very nice that there is Mixtral8x7b and Command R already but many people have 12-16gb vram GPUs, other 30B~ AQLM models would be very useful.

Thank you for such an amazing quantization method!