--- license: apache-2.0 datasets: - fblgit/simple-math - jondurbin/bagel-v0.3 library_name: transformers tags: - math - UNA - juanako --- # UNA-34BeagleSimpleMath-32K-v1 This is a fine-tuned version of fblgit/UNA-34Beagles-32K-v1 using fblgit/simple-math dataset. Powered by The Bagel v0.3 and Yi-34B Trained on AXOLOTL! ### 34BEAGLES MATH EVALS ``` |Tasks|Version| Filter |n-shot| Metric |Value | |Stderr| |-----|-------|----------|-----:|-----------|-----:|---|-----:| |gsm8k|Yaml |get-answer| 5|exact_match|0.6505|± |0.0131| | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |--------------|-------|------|-----:|--------|-----:|---|-----:| |arc_challenge |Yaml |none | 25|acc |0.7090|± |0.0133| | | |none | 25|acc_norm|0.7329|± |0.0129| |truthfulqa_mc2|Yaml |none | 0|acc |0.7378|± |0.0141| | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |---------------------------------------|-------|------|-----:|------|-----:|---|-----:| |mmlu |N/A |none | 0|acc |0.7524|± |0.1045| | - humanities |N/A |none | 0|acc |0.7307|± |0.0846| | - formal_logic |Yaml |none | 0|acc |0.5873|± |0.0440| | - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265| | - high_school_us_history |Yaml |none | 0|acc |0.9167|± |0.0194| | - high_school_world_history |Yaml |none | 0|acc |0.9114|± |0.0185| | - international_law |Yaml |none | 0|acc |0.8430|± |0.0332| | - jurisprudence |Yaml |none | 0|acc |0.8704|± |0.0325| | - logical_fallacies |Yaml |none | 0|acc |0.8589|± |0.0274| | - moral_disputes |Yaml |none | 0|acc |0.7717|± |0.0226| | - moral_scenarios |Yaml |none | 0|acc |0.7374|± |0.0147| | - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227| | - prehistory |Yaml |none | 0|acc |0.8549|± |0.0196| | - professional_law |Yaml |none | 0|acc |0.5724|± |0.0126| | - world_religions |Yaml |none | 0|acc |0.8830|± |0.0246| | - other |N/A |none | 0|acc |0.7937|± |0.1029| | - business_ethics |Yaml |none | 0|acc |0.7800|± |0.0416| | - clinical_knowledge |Yaml |none | 0|acc |0.8000|± |0.0246| | - college_medicine |Yaml |none | 0|acc |0.6936|± |0.0351| | - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500| | - human_aging |Yaml |none | 0|acc |0.7534|± |0.0289| | - management |Yaml |none | 0|acc |0.8447|± |0.0359| | - marketing |Yaml |none | 0|acc |0.9316|± |0.0165| | - medical_genetics |Yaml |none | 0|acc |0.8700|± |0.0338| | - miscellaneous |Yaml |none | 0|acc |0.8953|± |0.0109| | - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221| | - professional_accounting |Yaml |none | 0|acc |0.6277|± |0.0288| | - professional_medicine |Yaml |none | 0|acc |0.8015|± |0.0242| | - virology |Yaml |none | 0|acc |0.5723|± |0.0385| | - social_sciences |N/A |none | 0|acc |0.8274|± |0.0667| | - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458| | - high_school_geography |Yaml |none | 0|acc |0.8889|± |0.0224| | - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160| | - high_school_macroeconomics |Yaml |none | 0|acc |0.7897|± |0.0207| | - high_school_microeconomics |Yaml |none | 0|acc |0.8697|± |0.0219| | - high_school_psychology |Yaml |none | 0|acc |0.8899|± |0.0134| | - human_sexuality |Yaml |none | 0|acc |0.8550|± |0.0309| | - professional_psychology |Yaml |none | 0|acc |0.7745|± |0.0169| | - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439| | - security_studies |Yaml |none | 0|acc |0.7796|± |0.0265| | - sociology |Yaml |none | 0|acc |0.8657|± |0.0241| | - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314| | - stem |N/A |none | 0|acc |0.6708|± |0.1236| | - abstract_algebra |Yaml |none | 0|acc |0.4900|± |0.0502| | - anatomy |Yaml |none | 0|acc |0.7259|± |0.0385| | - astronomy |Yaml |none | 0|acc |0.8487|± |0.0292| | - college_biology |Yaml |none | 0|acc |0.8750|± |0.0277| | - college_chemistry |Yaml |none | 0|acc |0.5200|± |0.0502| | - college_computer_science |Yaml |none | 0|acc |0.6200|± |0.0488| | - college_mathematics |Yaml |none | 0|acc |0.4300|± |0.0498| | - college_physics |Yaml |none | 0|acc |0.5686|± |0.0493| | - computer_security |Yaml |none | 0|acc |0.7800|± |0.0416| | - conceptual_physics |Yaml |none | 0|acc |0.7404|± |0.0287| | - electrical_engineering |Yaml |none | 0|acc |0.7172|± |0.0375| | - elementary_mathematics |Yaml |none | 0|acc |0.6720|± |0.0242| | - high_school_biology |Yaml |none | 0|acc |0.9032|± |0.0168| | - high_school_chemistry |Yaml |none | 0|acc |0.6256|± |0.0341| | - high_school_computer_science |Yaml |none | 0|acc |0.7800|± |0.0416| | - high_school_mathematics |Yaml |none | 0|acc |0.4667|± |0.0304| | - high_school_physics |Yaml |none | 0|acc |0.5033|± |0.0408| | - high_school_statistics |Yaml |none | 0|acc |0.6435|± |0.0327| | - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472| | Groups |Version|Filter|n-shot|Metric|Value | |Stderr| |------------------|-------|------|-----:|------|-----:|---|-----:| |mmlu |N/A |none | 0|acc |0.7524|± |0.1045| | - humanities |N/A |none | 0|acc |0.7307|± |0.0846| | - other |N/A |none | 0|acc |0.7937|± |0.1029| | - social_sciences|N/A |none | 0|acc |0.8274|± |0.0667| | - stem |N/A |none | 0|acc |0.6708|± |0.1236| ``` ### 34BEAGLES (The Base Model) ``` | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |--------------|-------|------|-----:|--------|-----:|---|-----:| |arc_challenge |Yaml |none | 25|acc |0.7039|± |0.0133| | | |none | 25|acc_norm|0.7321|± |0.0129| |truthfulqa_mc2|Yaml |none | 0|acc |0.7387|± |0.0141| |Tasks|Version| Filter |n-shot| Metric |Value | |Stderr| |-----|-------|----------|-----:|-----------|-----:|---|-----:| |gsm8k|Yaml |get-answer| 5|exact_match|0.6399|± |0.0132| | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |---------------------------------------|-------|------|-----:|------|-----:|---|-----:| |mmlu |N/A |none | 0|acc |0.7477|± |0.1079| | - humanities |N/A |none | 0|acc |0.7188|± |0.0855| | - formal_logic |Yaml |none | 0|acc |0.5794|± |0.0442| | - high_school_european_history |Yaml |none | 0|acc |0.8667|± |0.0265| | - high_school_us_history |Yaml |none | 0|acc |0.9069|± |0.0204| | - high_school_world_history |Yaml |none | 0|acc |0.9072|± |0.0189| | - international_law |Yaml |none | 0|acc |0.8264|± |0.0346| | - jurisprudence |Yaml |none | 0|acc |0.8796|± |0.0315| | - logical_fallacies |Yaml |none | 0|acc |0.8405|± |0.0288| | - moral_disputes |Yaml |none | 0|acc |0.7746|± |0.0225| | - moral_scenarios |Yaml |none | 0|acc |0.6972|± |0.0154| | - philosophy |Yaml |none | 0|acc |0.8006|± |0.0227| | - prehistory |Yaml |none | 0|acc |0.8580|± |0.0194| | - professional_law |Yaml |none | 0|acc |0.5645|± |0.0127| | - world_religions |Yaml |none | 0|acc |0.8713|± |0.0257| | - other |N/A |none | 0|acc |0.7950|± |0.1057| | - business_ethics |Yaml |none | 0|acc |0.7700|± |0.0423| | - clinical_knowledge |Yaml |none | 0|acc |0.8038|± |0.0244| | - college_medicine |Yaml |none | 0|acc |0.7110|± |0.0346| | - global_facts |Yaml |none | 0|acc |0.5500|± |0.0500| | - human_aging |Yaml |none | 0|acc |0.7265|± |0.0299| | - management |Yaml |none | 0|acc |0.8544|± |0.0349| | - marketing |Yaml |none | 0|acc |0.9444|± |0.0150| | - medical_genetics |Yaml |none | 0|acc |0.8800|± |0.0327| | - miscellaneous |Yaml |none | 0|acc |0.8978|± |0.0108| | - nutrition |Yaml |none | 0|acc |0.8170|± |0.0221| | - professional_accounting |Yaml |none | 0|acc |0.6312|± |0.0288| | - professional_medicine |Yaml |none | 0|acc |0.8051|± |0.0241| | - virology |Yaml |none | 0|acc |0.5602|± |0.0386| | - social_sciences |N/A |none | 0|acc |0.8297|± |0.0664| | - econometrics |Yaml |none | 0|acc |0.6140|± |0.0458| | - high_school_geography |Yaml |none | 0|acc |0.8939|± |0.0219| | - high_school_government_and_politics|Yaml |none | 0|acc |0.9482|± |0.0160| | - high_school_macroeconomics |Yaml |none | 0|acc |0.7974|± |0.0204| | - high_school_microeconomics |Yaml |none | 0|acc |0.8655|± |0.0222| | - high_school_psychology |Yaml |none | 0|acc |0.8936|± |0.0132| | - human_sexuality |Yaml |none | 0|acc |0.8473|± |0.0315| | - professional_psychology |Yaml |none | 0|acc |0.7778|± |0.0168| | - public_relations |Yaml |none | 0|acc |0.7000|± |0.0439| | - security_studies |Yaml |none | 0|acc |0.7837|± |0.0264| | - sociology |Yaml |none | 0|acc |0.8657|± |0.0241| | - us_foreign_policy |Yaml |none | 0|acc |0.8900|± |0.0314| | - stem |N/A |none | 0|acc |0.6641|± |0.1291| | - abstract_algebra |Yaml |none | 0|acc |0.4800|± |0.0502| | - anatomy |Yaml |none | 0|acc |0.7407|± |0.0379| | - astronomy |Yaml |none | 0|acc |0.8618|± |0.0281| | - college_biology |Yaml |none | 0|acc |0.8611|± |0.0289| | - college_chemistry |Yaml |none | 0|acc |0.5300|± |0.0502| | - college_computer_science |Yaml |none | 0|acc |0.6100|± |0.0490| | - college_mathematics |Yaml |none | 0|acc |0.3800|± |0.0488| | - college_physics |Yaml |none | 0|acc |0.5588|± |0.0494| | - computer_security |Yaml |none | 0|acc |0.8000|± |0.0402| | - conceptual_physics |Yaml |none | 0|acc |0.7319|± |0.0290| | - electrical_engineering |Yaml |none | 0|acc |0.7034|± |0.0381| | - elementary_mathematics |Yaml |none | 0|acc |0.6587|± |0.0244| | - high_school_biology |Yaml |none | 0|acc |0.8935|± |0.0175| | - high_school_chemistry |Yaml |none | 0|acc |0.6305|± |0.0340| | - high_school_computer_science |Yaml |none | 0|acc |0.7700|± |0.0423| | - high_school_mathematics |Yaml |none | 0|acc |0.4296|± |0.0302| | - high_school_physics |Yaml |none | 0|acc |0.5166|± |0.0408| | - high_school_statistics |Yaml |none | 0|acc |0.6528|± |0.0325| | - machine_learning |Yaml |none | 0|acc |0.5536|± |0.0472| | Groups |Version|Filter|n-shot|Metric|Value | |Stderr| |------------------|-------|------|-----:|------|-----:|---|-----:| |mmlu |N/A |none | 0|acc |0.7477|± |0.1079| | - humanities |N/A |none | 0|acc |0.7188|± |0.0855| | - other |N/A |none | 0|acc |0.7950|± |0.1057| | - social_sciences|N/A |none | 0|acc |0.8297|± |0.0664| | - stem |N/A |none | 0|acc |0.6641|± |0.1291| ``` So I guess, SimpleMath: 2+2=4 4-1=3 SIMPLE-MATH .. works! :)