File size: 1,585 Bytes
afde5d4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
---
license: cc-by-nc-4.0
datasets:
- fdqerq22ds/MathScaleQA-2M
---
## Overview
This is a reproduced MathScale-Mistral model by finetuning the [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on our reproduced [MathScaleQA-2M](https://huggingface.co/datasets/fdqerq22ds/MathScaleQA-2M) dataset, following the hyperparameters in their [original paper](https://arxiv.org/abs/2403.02884) to ensure the effectiveness of our reproduction.
## Reproduction Details
Fortunately, the reproduction was smooth, and we managed to match the reported performance metrics when evaluating on their [MWPBench](https://github.com/microsoft/unilm/tree/master/mathscale/MWPBench). Below, we present a comparison between the performance of their official model and our reproduced model:
| Model | GSM8K | MATH | CollegeMath | TAL | Math23k | Ape210k | GaokaoBench-Math | AGIE-Gaokao-Math | AGIE-SAT-Math | AGIE-MATH | MicroAverage | MacroAverage |
|-------------------------------|-------|------|-------------|------|---------|---------|------------------|------------------|---------------|-----------|--------------|--------------|
| Official MathScale-Mistral | 74.8 | 35.2 | 21.8 | 39.9 | 64.4 | 46.0 | 21.4 | 14.3 | 57.8 | 32.9 | 38.7 | 40.8 |
| [Reproduced MathScale-Mistral](https://huggingface.co/fdqerq22ds/MathScale-Mistral) | 74.0 | 34.5 | 22.0 | 39.6 | 61.7 | 45.1 | 21.6 | 15.5 | 56.8 | 34.4 | 38.3 | 40.5 | |