metadata
tags:
- gptq
- 4bit
- int4
- gptqmodel
- modelcloud
- llama-3.1
- 405b
- instruct
license: llama3.1
This model has been quantized using GPTQModel.
- bits: 4
- group_size: 128
- desc_act: true
- static_groups: false
- sym: true
- lm_head: false
- damp_percent: 0.01
- true_sequential: true
- model_name_or_path: ""
- model_file_base_name: "model"
- quant_method: "gptq"
- checkpoint_format: "gptq"
- meta：
- quantizer: "gptqmodel:0.9.9-dev0"
Example:
from transformers import AutoTokenizer
from gptqmodel import GPTQModel
model_name = "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit"
prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = GPTQModel.from_quantized(model_name)
input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
lm-eval benchmark
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
|arc_challenge | 1|none | 0|acc |↑ |0.5990|± |0.0143|
| | |none | 0|acc_norm |↑ |0.6425|± |0.0140|
|arc_easy | 1|none | 0|acc |↑ |0.8645|± |0.0070|
| | |none | 0|acc_norm |↑ |0.8359|± |0.0076|
|boolq | 2|none | 0|acc |↑ |0.8820|± |0.0056|
|hellaswag | 1|none | 0|acc |↑ |0.6560|± |0.0047|
| | |none | 0|acc_norm |↑ |0.8446|± |0.0036|
|lambada_openai | 1|none | 0|acc |↑ |0.7252|± |0.0062|
| | |none | 0|perplexity|↓ |3.5576|± |0.0880|
|mmlu | 1|none | |acc |↑ |0.8245|± |0.0031|
| - humanities | 1|none | |acc |↑ |0.7892|± |0.0058|
| - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431|
| - high_school_european_history | 0|none | 0|acc |↑ |0.8667|± |0.0265|
| - high_school_us_history | 0|none | 0|acc |↑ |0.9314|± |0.0177|
| - high_school_world_history | 0|none | 0|acc |↑ |0.9367|± |0.0158|
| - international_law | 0|none | 0|acc |↑ |0.9091|± |0.0262|
| - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315|
| - logical_fallacies | 0|none | 0|acc |↑ |0.8834|± |0.0252|
| - moral_disputes | 0|none | 0|acc |↑ |0.8295|± |0.0202|
| - moral_scenarios | 0|none | 0|acc |↑ |0.7888|± |0.0137|
| - philosophy | 0|none | 0|acc |↑ |0.8521|± |0.0202|
| - prehistory | 0|none | 0|acc |↑ |0.8735|± |0.0185|
| - professional_law | 0|none | 0|acc |↑ |0.6760|± |0.0120|
| - world_religions | 0|none | 0|acc |↑ |0.8830|± |0.0246|
| - other | 1|none | |acc |↑ |0.8539|± |0.0060|
| - business_ethics | 0|none | 0|acc |↑ |0.8100|± |0.0394|
| - clinical_knowledge | 0|none | 0|acc |↑ |0.8679|± |0.0208|
| - college_medicine | 0|none | 0|acc |↑ |0.7688|± |0.0321|
| - global_facts | 0|none | 0|acc |↑ |0.7000|± |0.0461|
| - human_aging | 0|none | 0|acc |↑ |0.8341|± |0.0250|
| - management | 0|none | 0|acc |↑ |0.8932|± |0.0306|
| - marketing | 0|none | 0|acc |↑ |0.9444|± |0.0150|
| - medical_genetics | 0|none | 0|acc |↑ |0.9300|± |0.0256|
| - miscellaneous | 0|none | 0|acc |↑ |0.9425|± |0.0083|
| - nutrition | 0|none | 0|acc |↑ |0.8987|± |0.0173|
| - professional_accounting | 0|none | 0|acc |↑ |0.6773|± |0.0279|
| - professional_medicine | 0|none | 0|acc |↑ |0.9228|± |0.0162|
| - virology | 0|none | 0|acc |↑ |0.5542|± |0.0387|
| - social sciences | 1|none | |acc |↑ |0.8833|± |0.0057|
| - econometrics | 0|none | 0|acc |↑ |0.7193|± |0.0423|
| - high_school_geography | 0|none | 0|acc |↑ |0.9394|± |0.0170|
| - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115|
| - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8615|± |0.0175|
| - high_school_microeconomics | 0|none | 0|acc |↑ |0.9412|± |0.0153|
| - high_school_psychology | 0|none | 0|acc |↑ |0.9321|± |0.0108|
| - human_sexuality | 0|none | 0|acc |↑ |0.8550|± |0.0309|
| - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145|
| - public_relations | 0|none | 0|acc |↑ |0.7636|± |0.0407|
| - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248|
| - sociology | 0|none | 0|acc |↑ |0.9204|± |0.0191|
| - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256|
| - stem | 1|none | |acc |↑ |0.7907|± |0.0070|
| - abstract_algebra | 0|none | 0|acc |↑ |0.5800|± |0.0496|
| - anatomy | 0|none | 0|acc |↑ |0.8296|± |0.0325|
| - astronomy | 0|none | 0|acc |↑ |0.9145|± |0.0228|
| - college_biology | 0|none | 0|acc |↑ |0.9236|± |0.0222|
| - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496|
| - college_computer_science | 0|none | 0|acc |↑ |0.7300|± |0.0446|
| - college_mathematics | 0|none | 0|acc |↑ |0.5800|± |0.0496|
| - college_physics | 0|none | 0|acc |↑ |0.7157|± |0.0449|
| - computer_security | 0|none | 0|acc |↑ |0.8000|± |0.0402|
| - conceptual_physics | 0|none | 0|acc |↑ |0.8383|± |0.0241|
| - electrical_engineering | 0|none | 0|acc |↑ |0.7931|± |0.0338|
| - elementary_mathematics | 0|none | 0|acc |↑ |0.8730|± |0.0171|
| - high_school_biology | 0|none | 0|acc |↑ |0.9161|± |0.0158|
| - high_school_chemistry | 0|none | 0|acc |↑ |0.7685|± |0.0297|
| - high_school_computer_science | 0|none | 0|acc |↑ |0.9600|± |0.0197|
| - high_school_mathematics | 0|none | 0|acc |↑ |0.6556|± |0.0290|
| - high_school_physics | 0|none | 0|acc |↑ |0.7086|± |0.0371|
| - high_school_statistics | 0|none | 0|acc |↑ |0.7778|± |0.0284|
| - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433|
|openbookqa | 1|none | 0|acc |↑ |0.3300|± |0.0210|
| | |none | 0|acc_norm |↑ |0.4420|± |0.0222|
|piqa | 1|none | 0|acc |↑ |0.8188|± |0.0090|
| | |none | 0|acc_norm |↑ |0.8308|± |0.0087|
|rte | 1|none | 0|acc |↑ |0.7220|± |0.0270|
|truthfulqa_mc1 | 2|none | 0|acc |↑ |0.4333|± |0.0173|
|winogrande | 1|none | 0|acc |↑ |0.7656|± |0.0119|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu | 1|none | |acc |↑ |0.8245|± |0.0031|
| - humanities | 1|none | |acc |↑ |0.7892|± |0.0058|
| - other | 1|none | |acc |↑ |0.8539|± |0.0060|
| - social sciences| 1|none | |acc |↑ |0.8833|± |0.0057|
| - stem | 1|none | |acc |↑ |0.7907|± |0.0070|