Official AQLM quantization of mistralai/Mistral-7B-Instruct-v0.2 .

For this quantization, we used 2 codebooks of 8 bits.

Results:

Model	Quantization	MMLU (5-shot)	Model size, Gb
mistralai/Mistral-7B-Instruct-v0.2	None	0.5912	14.5
	2x8	0.4384	2.3

Downloads last month: 66

Safetensors

Model size

2.01B params

Tensor type

FP16

Inference Examples

Text Generation

This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Adapters

1 model

Collection including ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

AQLM

Collection

AQLM quantized LLMs • 20 items • Updated May 3 • 44