metadata
license: apache-2.0
datasets:
- fblgit/simple-math
- jondurbin/bagel-v0.3
library_name: transformers
tags:
- math
- UNA
- juanako
UNA-34BeagleSimpleMath-32K-v1
This is a fine-tuned version of fblgit/UNA-34Beagles-32K-v1 using fblgit/simple-math dataset. Powered by The Bagel v0.3 and Yi-34B
Trained on AXOLOTL!
34BEAGLES MATH EVALS
|Tasks|Version| Filter |n-shot| Metric |Value | |Stderr|
|-----|-------|----------|-----:|-----------|-----:|---|-----:|
|gsm8k|Yaml |get-answer| 5|exact_match|0.6505|± |0.0131|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|--------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge |Yaml |none | 25|acc |0.7090|± |0.0133|
| | |none | 25|acc_norm|0.7329|± |0.0129|
|truthfulqa_mc2|Yaml |none | 0|acc |0.7378|± |0.0141|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.7524|± |0.1045|
| - humanities |N/A |none | 0|acc |0.7307|± |0.0846|
| - formal_logic |Yaml |none | 0|acc |0.5873|± |0.0440|
| - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265|
| - high_school_us_history |Yaml |none | 0|acc |0.9167|± |0.0194|
| - high_school_world_history |Yaml |none | 0|acc |0.9114|± |0.0185|
| - international_law |Yaml |none | 0|acc |0.8430|± |0.0332|
| - jurisprudence |Yaml |none | 0|acc |0.8704|± |0.0325|
| - logical_fallacies |Yaml |none | 0|acc |0.8589|± |0.0274|
| - moral_disputes |Yaml |none | 0|acc |0.7717|± |0.0226|
| - moral_scenarios |Yaml |none | 0|acc |0.7374|± |0.0147|
| - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227|
| - prehistory |Yaml |none | 0|acc |0.8549|± |0.0196|
| - professional_law |Yaml |none | 0|acc |0.5724|± |0.0126|
| - world_religions |Yaml |none | 0|acc |0.8830|± |0.0246|
| - other |N/A |none | 0|acc |0.7937|± |0.1029|
| - business_ethics |Yaml |none | 0|acc |0.7800|± |0.0416|
| - clinical_knowledge |Yaml |none | 0|acc |0.8000|± |0.0246|
| - college_medicine |Yaml |none | 0|acc |0.6936|± |0.0351|
| - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500|
| - human_aging |Yaml |none | 0|acc |0.7534|± |0.0289|
| - management |Yaml |none | 0|acc |0.8447|± |0.0359|
| - marketing |Yaml |none | 0|acc |0.9316|± |0.0165|
| - medical_genetics |Yaml |none | 0|acc |0.8700|± |0.0338|
| - miscellaneous |Yaml |none | 0|acc |0.8953|± |0.0109|
| - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221|
| - professional_accounting |Yaml |none | 0|acc |0.6277|± |0.0288|
| - professional_medicine |Yaml |none | 0|acc |0.8015|± |0.0242|
| - virology |Yaml |none | 0|acc |0.5723|± |0.0385|
| - social_sciences |N/A |none | 0|acc |0.8274|± |0.0667|
| - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458|
| - high_school_geography |Yaml |none | 0|acc |0.8889|± |0.0224|
| - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160|
| - high_school_macroeconomics |Yaml |none | 0|acc |0.7897|± |0.0207|
| - high_school_microeconomics |Yaml |none | 0|acc |0.8697|± |0.0219|
| - high_school_psychology |Yaml |none | 0|acc |0.8899|± |0.0134|
| - human_sexuality |Yaml |none | 0|acc |0.8550|± |0.0309|
| - professional_psychology |Yaml |none | 0|acc |0.7745|± |0.0169|
| - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439|
| - security_studies |Yaml |none | 0|acc |0.7796|± |0.0265|
| - sociology |Yaml |none | 0|acc |0.8657|± |0.0241|
| - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314|
| - stem |N/A |none | 0|acc |0.6708|± |0.1236|
| - abstract_algebra |Yaml |none | 0|acc |0.4900|± |0.0502|
| - anatomy |Yaml |none | 0|acc |0.7259|± |0.0385|
| - astronomy |Yaml |none | 0|acc |0.8487|± |0.0292|
| - college_biology |Yaml |none | 0|acc |0.8750|± |0.0277|
| - college_chemistry |Yaml |none | 0|acc |0.5200|± |0.0502|
| - college_computer_science |Yaml |none | 0|acc |0.6200|± |0.0488|
| - college_mathematics |Yaml |none | 0|acc |0.4300|± |0.0498|
| - college_physics |Yaml |none | 0|acc |0.5686|± |0.0493|
| - computer_security |Yaml |none | 0|acc |0.7800|± |0.0416|
| - conceptual_physics |Yaml |none | 0|acc |0.7404|± |0.0287|
| - electrical_engineering |Yaml |none | 0|acc |0.7172|± |0.0375|
| - elementary_mathematics |Yaml |none | 0|acc |0.6720|± |0.0242|
| - high_school_biology |Yaml |none | 0|acc |0.9032|± |0.0168|
| - high_school_chemistry |Yaml |none | 0|acc |0.6256|± |0.0341|
| - high_school_computer_science |Yaml |none | 0|acc |0.7800|± |0.0416|
| - high_school_mathematics |Yaml |none | 0|acc |0.4667|± |0.0304|
| - high_school_physics |Yaml |none | 0|acc |0.5033|± |0.0408|
| - high_school_statistics |Yaml |none | 0|acc |0.6435|± |0.0327|
| - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472|
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.7524|± |0.1045|
| - humanities |N/A |none | 0|acc |0.7307|± |0.0846|
| - other |N/A |none | 0|acc |0.7937|± |0.1029|
| - social_sciences|N/A |none | 0|acc |0.8274|± |0.0667|
| - stem |N/A |none | 0|acc |0.6708|± |0.1236|
34BEAGLES (The Base Model)
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|--------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge |Yaml |none | 25|acc |0.7039|± |0.0133|
| | |none | 25|acc_norm|0.7321|± |0.0129|
|truthfulqa_mc2|Yaml |none | 0|acc |0.7387|± |0.0141|
|Tasks|Version| Filter |n-shot| Metric |Value | |Stderr|
|-----|-------|----------|-----:|-----------|-----:|---|-----:|
|gsm8k|Yaml |get-answer| 5|exact_match|0.6399|± |0.0132|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.7477|± |0.1079|
| - humanities |N/A |none | 0|acc |0.7188|± |0.0855|
| - formal_logic |Yaml |none | 0|acc |0.5794|± |0.0442|
| - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265|
| - high_school_us_history |Yaml |none | 0|acc |0.9069|± |0.0204|
| - high_school_world_history |Yaml |none | 0|acc |0.9072|± |0.0189|
| - international_law |Yaml |none | 0|acc |0.8264|± |0.0346|
| - jurisprudence |Yaml |none | 0|acc |0.8796|± |0.0315|
| - logical_fallacies |Yaml |none | 0|acc |0.8405|± |0.0288|
| - moral_disputes |Yaml |none | 0|acc |0.7746|± |0.0225|
| - moral_scenarios |Yaml |none | 0|acc |0.6972|± |0.0154|
| - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227|
| - prehistory |Yaml |none | 0|acc |0.8580|± |0.0194|
| - professional_law |Yaml |none | 0|acc |0.5645|± |0.0127|
| - world_religions |Yaml |none | 0|acc |0.8713|± |0.0257|
| - other |N/A |none | 0|acc |0.7950|± |0.1057|
| - business_ethics |Yaml |none | 0|acc |0.7700|± |0.0423|
| - clinical_knowledge |Yaml |none | 0|acc |0.8038|± |0.0244|
| - college_medicine |Yaml |none | 0|acc |0.7110|± |0.0346|
| - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500|
| - human_aging |Yaml |none | 0|acc |0.7265|± |0.0299|
| - management |Yaml |none | 0|acc |0.8544|± |0.0349|
| - marketing |Yaml |none | 0|acc |0.9444|± |0.0150|
| - medical_genetics |Yaml |none | 0|acc |0.8800|± |0.0327|
| - miscellaneous |Yaml |none | 0|acc |0.8978|± |0.0108|
| - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221|
| - professional_accounting |Yaml |none | 0|acc |0.6312|± |0.0288|
| - professional_medicine |Yaml |none | 0|acc |0.8051|± |0.0241|
| - virology |Yaml |none | 0|acc |0.5602|± |0.0386|
| - social_sciences |N/A |none | 0|acc |0.8297|± |0.0664|
| - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458|
| - high_school_geography |Yaml |none | 0|acc |0.8939|± |0.0219|
| - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160|
| - high_school_macroeconomics |Yaml |none | 0|acc |0.7974|± |0.0204|
| - high_school_microeconomics |Yaml |none | 0|acc |0.8655|± |0.0222|
| - high_school_psychology |Yaml |none | 0|acc |0.8936|± |0.0132|
| - human_sexuality |Yaml |none | 0|acc |0.8473|± |0.0315|
| - professional_psychology |Yaml |none | 0|acc |0.7778|± |0.0168|
| - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439|
| - security_studies |Yaml |none | 0|acc |0.7837|± |0.0264|
| - sociology |Yaml |none | 0|acc |0.8657|± |0.0241|
| - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314|
| - stem |N/A |none | 0|acc |0.6641|± |0.1291|
| - abstract_algebra |Yaml |none | 0|acc |0.4800|± |0.0502|
| - anatomy |Yaml |none | 0|acc |0.7407|± |0.0379|
| - astronomy |Yaml |none | 0|acc |0.8618|± |0.0281|
| - college_biology |Yaml |none | 0|acc |0.8611|± |0.0289|
| - college_chemistry |Yaml |none | 0|acc |0.5300|± |0.0502|
| - college_computer_science |Yaml |none | 0|acc |0.6100|± |0.0490|
| - college_mathematics |Yaml |none | 0|acc |0.3800|± |0.0488|
| - college_physics |Yaml |none | 0|acc |0.5588|± |0.0494|
| - computer_security |Yaml |none | 0|acc |0.8000|± |0.0402|
| - conceptual_physics |Yaml |none | 0|acc |0.7319|± |0.0290|
| - electrical_engineering |Yaml |none | 0|acc |0.7034|± |0.0381|
| - elementary_mathematics |Yaml |none | 0|acc |0.6587|± |0.0244|
| - high_school_biology |Yaml |none | 0|acc |0.8935|± |0.0175|
| - high_school_chemistry |Yaml |none | 0|acc |0.6305|± |0.0340|
| - high_school_computer_science |Yaml |none | 0|acc |0.7700|± |0.0423|
| - high_school_mathematics |Yaml |none | 0|acc |0.4296|± |0.0302|
| - high_school_physics |Yaml |none | 0|acc |0.5166|± |0.0408|
| - high_school_statistics |Yaml |none | 0|acc |0.6528|± |0.0325|
| - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472|
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.7477|± |0.1079|
| - humanities |N/A |none | 0|acc |0.7188|± |0.0855|
| - other |N/A |none | 0|acc |0.7950|± |0.1057|
| - social_sciences|N/A |none | 0|acc |0.8297|± |0.0664|
| - stem |N/A |none | 0|acc |0.6641|± |0.1291|
So I guess, SimpleMath: 2+2=4 4-1=3 SIMPLE-MATH .. works! :)