Hugging Face's logo

ModelCloud
/
Meta-Llama-3.1-405B-Instruct-gptq-4bit

Text Generation
Transformers
Safetensors
llama
gptq
4bit
int4
gptqmodel
modelcloud
llama-3.1
405b
instruct
conversational
text-generation-inference
Inference Endpoints
4-bit precision
Model card Files Community
1
lrl-modelcloud commited on
Commit
53e7b6f
•
1 Parent(s): 14916f0

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +140 -0
README.md ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - gptq
4
+ - 4bit
5
+ - int4
6
+ - gptqmodel
7
+ - modelcloud
8
+ - llama-3.1
9
+ - 405b
10
+ - instruct
11
+ license: llama3.1
12
+ ---
13
+ This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel).
14
+
15
+ - **bits**: 4
16
+ - **group_size**: 128
17
+ - **desc_act**: true
18
+ - **static_groups**: false
19
+ - **sym**: true
20
+ - **lm_head**: false
21
+ - **damp_percent**: 0.01
22
+ - **true_sequential**: true
23
+ - **model_name_or_path**: ""
24
+ - **model_file_base_name**: "model"
25
+ - **quant_method**: "gptq"
26
+ - **checkpoint_format**: "gptq"
27
+ - **meta**：
28
+ - **quantizer**: "gptqmodel:0.9.9-dev0"
29
+
30
+ ## Example:
31
+ ```python
32
+ from transformers import AutoTokenizer
33
+ from gptqmodel import GPTQModel
34
+
35
+ model_name = "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit"
36
+
37
+ prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]
38
+
39
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
40
+
41
+ model = GPTQModel.from_quantized(model_name)
42
+
43
+ input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
44
+ outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
45
+ result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
46
+
47
+ print(result)
48
+ ```
49
+
50
+ ## lm-eval benchmark
51
+ ```
52
+ | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
53
+ |---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
54
+ |arc_challenge | 1|none | 0|acc |↑ |0.5990|± |0.0143|
55
+ | | |none | 0|acc_norm |↑ |0.6425|± |0.0140|
56
+ |arc_easy | 1|none | 0|acc |↑ |0.8645|± |0.0070|
57
+ | | |none | 0|acc_norm |↑ |0.8359|± |0.0076|
58
+ |boolq | 2|none | 0|acc |↑ |0.8820|± |0.0056|
59
+ |hellaswag | 1|none | 0|acc |↑ |0.6560|± |0.0047|
60
+ | | |none | 0|acc_norm |↑ |0.8446|± |0.0036|
61
+ |lambada_openai | 1|none | 0|acc |↑ |0.7252|± |0.0062|
62
+ | | |none | 0|perplexity|↓ |3.5576|± |0.0880|
63
+ |mmlu | 1|none | |acc |↑ |0.8245|± |0.0031|
64
+ | - humanities | 1|none | |acc |↑ |0.7892|± |0.0058|
65
+ | - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431|
66
+ | - high_school_european_history | 0|none | 0|acc |↑ |0.8667|± |0.0265|
67
+ | - high_school_us_history | 0|none | 0|acc |↑ |0.9314|± |0.0177|
68
+ | - high_school_world_history | 0|none | 0|acc |↑ |0.9367|± |0.0158|
69
+ | - international_law | 0|none | 0|acc |↑ |0.9091|± |0.0262|
70
+ | - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315|
71
+ | - logical_fallacies | 0|none | 0|acc |↑ |0.8834|± |0.0252|
72
+ | - moral_disputes | 0|none | 0|acc |↑ |0.8295|± |0.0202|
73
+ | - moral_scenarios | 0|none | 0|acc |↑ |0.7888|± |0.0137|
74
+ | - philosophy | 0|none | 0|acc |↑ |0.8521|± |0.0202|
75
+ | - prehistory | 0|none | 0|acc |↑ |0.8735|± |0.0185|
76
+ | - professional_law | 0|none | 0|acc |↑ |0.6760|± |0.0120|
77
+ | - world_religions | 0|none | 0|acc |↑ |0.8830|± |0.0246|
78
+ | - other | 1|none | |acc |↑ |0.8539|± |0.0060|
79
+ | - business_ethics | 0|none | 0|acc |↑ |0.8100|± |0.0394|
80
+ | - clinical_knowledge | 0|none | 0|acc |↑ |0.8679|± |0.0208|
81
+ | - college_medicine | 0|none | 0|acc |↑ |0.7688|± |0.0321|
82
+ | - global_facts | 0|none | 0|acc |↑ |0.7000|± |0.0461|
83
+ | - human_aging | 0|none | 0|acc |↑ |0.8341|± |0.0250|
84
+ | - management | 0|none | 0|acc |↑ |0.8932|± |0.0306|
85
+ | - marketing | 0|none | 0|acc |↑ |0.9444|± |0.0150|
86
+ | - medical_genetics | 0|none | 0|acc |↑ |0.9300|± |0.0256|
87
+ | - miscellaneous | 0|none | 0|acc |↑ |0.9425|± |0.0083|
88
+ | - nutrition | 0|none | 0|acc |↑ |0.8987|± |0.0173|
89
+ | - professional_accounting | 0|none | 0|acc |↑ |0.6773|± |0.0279|
90
+ | - professional_medicine | 0|none | 0|acc |↑ |0.9228|± |0.0162|
91
+ | - virology | 0|none | 0|acc |↑ |0.5542|± |0.0387|
92
+ | - social sciences | 1|none | |acc |↑ |0.8833|± |0.0057|
93
+ | - econometrics | 0|none | 0|acc |↑ |0.7193|± |0.0423|
94
+ | - high_school_geography | 0|none | 0|acc |↑ |0.9394|± |0.0170|
95
+ | - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115|
96
+ | - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8615|± |0.0175|
97
+ | - high_school_microeconomics | 0|none | 0|acc |↑ |0.9412|± |0.0153|
98
+ | - high_school_psychology | 0|none | 0|acc |↑ |0.9321|± |0.0108|
99
+ | - human_sexuality | 0|none | 0|acc |↑ |0.8550|± |0.0309|
100
+ | - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145|
101
+ | - public_relations | 0|none | 0|acc |↑ |0.7636|± |0.0407|
102
+ | - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248|
103
+ | - sociology | 0|none | 0|acc |↑ |0.9204|± |0.0191|
104
+ | - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256|
105
+ | - stem | 1|none | |acc |↑ |0.7907|± |0.0070|
106
+ | - abstract_algebra | 0|none | 0|acc |↑ |0.5800|± |0.0496|
107
+ | - anatomy | 0|none | 0|acc |↑ |0.8296|± |0.0325|
108
+ | - astronomy | 0|none | 0|acc |↑ |0.9145|± |0.0228|
109
+ | - college_biology | 0|none | 0|acc |↑ |0.9236|± |0.0222|
110
+ | - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496|
111
+ | - college_computer_science | 0|none | 0|acc |↑ |0.7300|± |0.0446|
112
+ | - college_mathematics | 0|none | 0|acc |↑ |0.5800|± |0.0496|
113
+ | - college_physics | 0|none | 0|acc |↑ |0.7157|± |0.0449|
114
+ | - computer_security | 0|none | 0|acc |↑ |0.8000|± |0.0402|
115
+ | - conceptual_physics | 0|none | 0|acc |↑ |0.8383|± |0.0241|
116
+ | - electrical_engineering | 0|none | 0|acc |↑ |0.7931|± |0.0338|
117
+ | - elementary_mathematics | 0|none | 0|acc |↑ |0.8730|± |0.0171|
118
+ | - high_school_biology | 0|none | 0|acc |↑ |0.9161|± |0.0158|
119
+ | - high_school_chemistry | 0|none | 0|acc |↑ |0.7685|± |0.0297|
120
+ | - high_school_computer_science | 0|none | 0|acc |↑ |0.9600|± |0.0197|
121
+ | - high_school_mathematics | 0|none | 0|acc |↑ |0.6556|± |0.0290|
122
+ | - high_school_physics | 0|none | 0|acc |↑ |0.7086|± |0.0371|
123
+ | - high_school_statistics | 0|none | 0|acc |↑ |0.7778|± |0.0284|
124
+ | - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433|
125
+ |openbookqa | 1|none | 0|acc |↑ |0.3300|± |0.0210|
126
+ | | |none | 0|acc_norm |↑ |0.4420|± |0.0222|
127
+ |piqa | 1|none | 0|acc |↑ |0.8188|± |0.0090|
128
+ | | |none | 0|acc_norm |↑ |0.8308|± |0.0087|
129
+ |rte | 1|none | 0|acc |↑ |0.7220|± |0.0270|
130
+ |truthfulqa_mc1 | 2|none | 0|acc |↑ |0.4333|± |0.0173|
131
+ |winogrande | 1|none | 0|acc |↑ |0.7656|± |0.0119|
132
+
133
+ | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
134
+ |------------------|------:|------|------|------|---|-----:|---|-----:|
135
+ |mmlu | 1|none | |acc |↑ |0.8245|± |0.0031|
136
+ | - humanities | 1|none | |acc |↑ |0.7892|± |0.0058|
137
+ | - other | 1|none | |acc |↑ |0.8539|± |0.0060|
138
+ | - social sciences| 1|none | |acc |↑ |0.8833|± |0.0057|
139
+ | - stem | 1|none | |acc |↑ |0.7907|± |0.0070|
140
+ ```