metadata
license: llama2
datasets:
- gair-prox/open-web-math-pro
language:
- en
pipeline_tag: text-generation
tags:
- llama2
- math
- reasoning
base_model:
- meta-llama/Llama-2-7b-hf
Llama-2-7B-ProXMath
ArXiv | Data: OpenWebMath-Pro | Code
Llama-2-7B-ProXMath is a math-adapted Llama-2-7B model that is continually pre-trained on OpenWebMath-Pro (a refined version by ProX) for 10B tokens.
Evaluations
ProX models are evaluated on 9 common math reasoning benchmarks.
| Model | asdiv | gsm8k | mathqa | mawps | minerva_math | mmlu_stem | sat_math | svamp | tabmwp | average |
|---|---|---|---|---|---|---|---|---|---|---|
| Llama-2-7B | 51.6 | 14.1 | 12.5 | 63.6 | 3.8 | 32.9 | 34.4 | 39.5 | 30.9 | 31.48 |
| Llama-2-7B-ProXMath | 63.7 | 30.6 | 40.1 | 79.3 | 16.8 | 43.8 | 53.1 | 50.2 | 37.3 | 46.1 |
Citation
@article{zhou2024programming,
title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
author={Zhou, Fan and Wang, Zengzhi and Liu, Qian and Li, Junlong and Liu, Pengfei},
journal={arXiv preprint arXiv:2409.17115},
year={2024}
}