Hugging Face's logo

fblgit
/
UNA-34BeagleSimpleMath-32K-v1

Text Generation
Transformers
Safetensors
llama
math
UNA
juanako
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
fblgit commited on
Commit
ead4b4a
•
1 Parent(s): f3298c3

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +191 -0
README.md ADDED
@@ -0,0 +1,191 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - fblgit/simple-math
5
+ - jondurbin/bagel-v0.3
6
+ library_name: transformers
7
+ tags:
8
+ - math
9
+ - UNA
10
+ - juanako
11
+ ---
12
+
13
+ # UNA-34BeagleSimpleMath-32K-v1
14
+
15
+ This is a fine-tuned version of fblgit/UNA-34Beagles-32K-v1 using fblgit/simple-math dataset. Powered by The Bagel v0.3 and Yi-34B
16
+
17
+ Trained on AXOLOTL!
18
+
19
+ ### 34BEAGLES MATH EVALS
20
+ ```
21
+ |Tasks|Version| Filter |n-shot| Metric |Value | |Stderr|
22
+ |-----|-------|----------|-----:|-----------|-----:|---|-----:|
23
+ |gsm8k|Yaml |get-answer| 5|exact_match|0.6505|± |0.0131|
24
+
25
+ | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
26
+ |--------------|-------|------|-----:|--------|-----:|---|-----:|
27
+ |arc_challenge |Yaml |none | 25|acc |0.7090|± |0.0133|
28
+ | | |none | 25|acc_norm|0.7329|± |0.0129|
29
+ |truthfulqa_mc2|Yaml |none | 0|acc |0.7378|± |0.0141|
30
+
31
+ | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
32
+ |---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
33
+ |mmlu |N/A |none | 0|acc |0.7524|± |0.1045|
34
+ | - humanities |N/A |none | 0|acc |0.7307|± |0.0846|
35
+ | - formal_logic |Yaml |none | 0|acc |0.5873|± |0.0440|
36
+ | - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265|
37
+ | - high_school_us_history |Yaml |none | 0|acc |0.9167|± |0.0194|
38
+ | - high_school_world_history |Yaml |none | 0|acc |0.9114|± |0.0185|
39
+ | - international_law |Yaml |none | 0|acc |0.8430|± |0.0332|
40
+ | - jurisprudence |Yaml |none | 0|acc |0.8704|± |0.0325|
41
+ | - logical_fallacies |Yaml |none | 0|acc |0.8589|± |0.0274|
42
+ | - moral_disputes |Yaml |none | 0|acc |0.7717|± |0.0226|
43
+ | - moral_scenarios |Yaml |none | 0|acc |0.7374|± |0.0147|
44
+ | - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227|
45
+ | - prehistory |Yaml |none | 0|acc |0.8549|± |0.0196|
46
+ | - professional_law |Yaml |none | 0|acc |0.5724|± |0.0126|
47
+ | - world_religions |Yaml |none | 0|acc |0.8830|± |0.0246|
48
+ | - other |N/A |none | 0|acc |0.7937|± |0.1029|
49
+ | - business_ethics |Yaml |none | 0|acc |0.7800|± |0.0416|
50
+ | - clinical_knowledge |Yaml |none | 0|acc |0.8000|± |0.0246|
51
+ | - college_medicine |Yaml |none | 0|acc |0.6936|± |0.0351|
52
+ | - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500|
53
+ | - human_aging |Yaml |none | 0|acc |0.7534|± |0.0289|
54
+ | - management |Yaml |none | 0|acc |0.8447|± |0.0359|
55
+ | - marketing |Yaml |none | 0|acc |0.9316|± |0.0165|
56
+ | - medical_genetics |Yaml |none | 0|acc |0.8700|± |0.0338|
57
+ | - miscellaneous |Yaml |none | 0|acc |0.8953|± |0.0109|
58
+ | - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221|
59
+ | - professional_accounting |Yaml |none | 0|acc |0.6277|± |0.0288|
60
+ | - professional_medicine |Yaml |none | 0|acc |0.8015|± |0.0242|
61
+ | - virology |Yaml |none | 0|acc |0.5723|± |0.0385|
62
+ | - social_sciences |N/A |none | 0|acc |0.8274|± |0.0667|
63
+ | - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458|
64
+ | - high_school_geography |Yaml |none | 0|acc |0.8889|± |0.0224|
65
+ | - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160|
66
+ | - high_school_macroeconomics |Yaml |none | 0|acc |0.7897|± |0.0207|
67
+ | - high_school_microeconomics |Yaml |none | 0|acc |0.8697|± |0.0219|
68
+ | - high_school_psychology |Yaml |none | 0|acc |0.8899|± |0.0134|
69
+ | - human_sexuality |Yaml |none | 0|acc |0.8550|± |0.0309|
70
+ | - professional_psychology |Yaml |none | 0|acc |0.7745|± |0.0169|
71
+ | - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439|
72
+ | - security_studies |Yaml |none | 0|acc |0.7796|± |0.0265|
73
+ | - sociology |Yaml |none | 0|acc |0.8657|± |0.0241|
74
+ | - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314|
75
+ | - stem |N/A |none | 0|acc |0.6708|± |0.1236|
76
+ | - abstract_algebra |Yaml |none | 0|acc |0.4900|± |0.0502|
77
+ | - anatomy |Yaml |none | 0|acc |0.7259|± |0.0385|
78
+ | - astronomy |Yaml |none | 0|acc |0.8487|± |0.0292|
79
+ | - college_biology |Yaml |none | 0|acc |0.8750|± |0.0277|
80
+ | - college_chemistry |Yaml |none | 0|acc |0.5200|± |0.0502|
81
+ | - college_computer_science |Yaml |none | 0|acc |0.6200|± |0.0488|
82
+ | - college_mathematics |Yaml |none | 0|acc |0.4300|± |0.0498|
83
+ | - college_physics |Yaml |none | 0|acc |0.5686|± |0.0493|
84
+ | - computer_security |Yaml |none | 0|acc |0.7800|± |0.0416|
85
+ | - conceptual_physics |Yaml |none | 0|acc |0.7404|± |0.0287|
86
+ | - electrical_engineering |Yaml |none | 0|acc |0.7172|± |0.0375|
87
+ | - elementary_mathematics |Yaml |none | 0|acc |0.6720|± |0.0242|
88
+ | - high_school_biology |Yaml |none | 0|acc |0.9032|± |0.0168|
89
+ | - high_school_chemistry |Yaml |none | 0|acc |0.6256|± |0.0341|
90
+ | - high_school_computer_science |Yaml |none | 0|acc |0.7800|± |0.0416|
91
+ | - high_school_mathematics |Yaml |none | 0|acc |0.4667|± |0.0304|
92
+ | - high_school_physics |Yaml |none | 0|acc |0.5033|± |0.0408|
93
+ | - high_school_statistics |Yaml |none | 0|acc |0.6435|± |0.0327|
94
+ | - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472|
95
+
96
+ | Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
97
+ |------------------|-------|------|-----:|------|-----:|---|-----:|
98
+ |mmlu |N/A |none | 0|acc |0.7524|± |0.1045|
99
+ | - humanities |N/A |none | 0|acc |0.7307|± |0.0846|
100
+ | - other |N/A |none | 0|acc |0.7937|± |0.1029|
101
+ | - social_sciences|N/A |none | 0|acc |0.8274|± |0.0667|
102
+ | - stem |N/A |none | 0|acc |0.6708|± |0.1236|
103
+ ```
104
+
105
+ ### 34BEAGLES (The Base Model)
106
+ ```
107
+ | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
108
+ |--------------|-------|------|-----:|--------|-----:|---|-----:|
109
+ |arc_challenge |Yaml |none | 25|acc |0.7039|± |0.0133|
110
+ | | |none | 25|acc_norm|0.7321|± |0.0129|
111
+ |truthfulqa_mc2|Yaml |none | 0|acc |0.7387|± |0.0141|
112
+
113
+ |Tasks|Version| Filter |n-shot| Metric |Value | |Stderr|
114
+ |-----|-------|----------|-----:|-----------|-----:|---|-----:|
115
+ |gsm8k|Yaml |get-answer| 5|exact_match|0.6399|± |0.0132|
116
+
117
+ | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
118
+ |---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
119
+ |mmlu |N/A |none | 0|acc |0.7477|± |0.1079|
120
+ | - humanities |N/A |none | 0|acc |0.7188|± |0.0855|
121
+ | - formal_logic |Yaml |none | 0|acc |0.5794|± |0.0442|
122
+ | - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265|
123
+ | - high_school_us_history |Yaml |none | 0|acc |0.9069|± |0.0204|
124
+ | - high_school_world_history |Yaml |none | 0|acc |0.9072|± |0.0189|
125
+ | - international_law |Yaml |none | 0|acc |0.8264|± |0.0346|
126
+ | - jurisprudence |Yaml |none | 0|acc |0.8796|± |0.0315|
127
+ | - logical_fallacies |Yaml |none | 0|acc |0.8405|± |0.0288|
128
+ | - moral_disputes |Yaml |none | 0|acc |0.7746|± |0.0225|
129
+ | - moral_scenarios |Yaml |none | 0|acc |0.6972|± |0.0154|
130
+ | - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227|
131
+ | - prehistory |Yaml |none | 0|acc |0.8580|± |0.0194|
132
+ | - professional_law |Yaml |none | 0|acc |0.5645|± |0.0127|
133
+ | - world_religions |Yaml |none | 0|acc |0.8713|± |0.0257|
134
+ | - other |N/A |none | 0|acc |0.7950|± |0.1057|
135
+ | - business_ethics |Yaml |none | 0|acc |0.7700|± |0.0423|
136
+ | - clinical_knowledge |Yaml |none | 0|acc |0.8038|± |0.0244|
137
+ | - college_medicine |Yaml |none | 0|acc |0.7110|± |0.0346|
138
+ | - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500|
139
+ | - human_aging |Yaml |none | 0|acc |0.7265|± |0.0299|
140
+ | - management |Yaml |none | 0|acc |0.8544|± |0.0349|
141
+ | - marketing |Yaml |none | 0|acc |0.9444|± |0.0150|
142
+ | - medical_genetics |Yaml |none | 0|acc |0.8800|± |0.0327|
143
+ | - miscellaneous |Yaml |none | 0|acc |0.8978|± |0.0108|
144
+ | - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221|
145
+ | - professional_accounting |Yaml |none | 0|acc |0.6312|± |0.0288|
146
+ | - professional_medicine |Yaml |none | 0|acc |0.8051|± |0.0241|
147
+ | - virology |Yaml |none | 0|acc |0.5602|± |0.0386|
148
+ | - social_sciences |N/A |none | 0|acc |0.8297|± |0.0664|
149
+ | - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458|
150
+ | - high_school_geography |Yaml |none | 0|acc |0.8939|± |0.0219|
151
+ | - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160|
152
+ | - high_school_macroeconomics |Yaml |none | 0|acc |0.7974|± |0.0204|
153
+ | - high_school_microeconomics |Yaml |none | 0|acc |0.8655|± |0.0222|
154
+ | - high_school_psychology |Yaml |none | 0|acc |0.8936|± |0.0132|
155
+ | - human_sexuality |Yaml |none | 0|acc |0.8473|± |0.0315|
156
+ | - professional_psychology |Yaml |none | 0|acc |0.7778|± |0.0168|
157
+ | - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439|
158
+ | - security_studies |Yaml |none | 0|acc |0.7837|± |0.0264|
159
+ | - sociology |Yaml |none | 0|acc |0.8657|± |0.0241|
160
+ | - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314|
161
+ | - stem |N/A |none | 0|acc |0.6641|± |0.1291|
162
+ | - abstract_algebra |Yaml |none | 0|acc |0.4800|± |0.0502|
163
+ | - anatomy |Yaml |none | 0|acc |0.7407|± |0.0379|
164
+ | - astronomy |Yaml |none | 0|acc |0.8618|± |0.0281|
165
+ | - college_biology |Yaml |none | 0|acc |0.8611|± |0.0289|
166
+ | - college_chemistry |Yaml |none | 0|acc |0.5300|± |0.0502|
167
+ | - college_computer_science |Yaml |none | 0|acc |0.6100|± |0.0490|
168
+ | - college_mathematics |Yaml |none | 0|acc |0.3800|± |0.0488|
169
+ | - college_physics |Yaml |none | 0|acc |0.5588|± |0.0494|
170
+ | - computer_security |Yaml |none | 0|acc |0.8000|± |0.0402|
171
+ | - conceptual_physics |Yaml |none | 0|acc |0.7319|± |0.0290|
172
+ | - electrical_engineering |Yaml |none | 0|acc |0.7034|± |0.0381|
173
+ | - elementary_mathematics |Yaml |none | 0|acc |0.6587|± |0.0244|
174
+ | - high_school_biology |Yaml |none | 0|acc |0.8935|± |0.0175|
175
+ | - high_school_chemistry |Yaml |none | 0|acc |0.6305|± |0.0340|
176
+ | - high_school_computer_science |Yaml |none | 0|acc |0.7700|± |0.0423|
177
+ | - high_school_mathematics |Yaml |none | 0|acc |0.4296|± |0.0302|
178
+ | - high_school_physics |Yaml |none | 0|acc |0.5166|± |0.0408|
179
+ | - high_school_statistics |Yaml |none | 0|acc |0.6528|± |0.0325|
180
+ | - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472|
181
+
182
+ | Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
183
+ |------------------|-------|------|-----:|------|-----:|---|-----:|
184
+ |mmlu |N/A |none | 0|acc |0.7477|± |0.1079|
185
+ | - humanities |N/A |none | 0|acc |0.7188|± |0.0855|
186
+ | - other |N/A |none | 0|acc |0.7950|± |0.1057|
187
+ | - social_sciences|N/A |none | 0|acc |0.8297|± |0.0664|
188
+ | - stem |N/A |none | 0|acc |0.6641|± |0.1291|
189
+ ```
190
+
191
+ So I guess, SimpleMath: 2+2=4 4-1=3 SIMPLE-MATH .. works! :)