|
--- |
|
license: mit |
|
tags: |
|
- nlp |
|
- math |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
## Rho-1: Not All Tokens Are What You Need |
|
|
|
|
|
The Rho-1 series are pretrained language models that utilize Selective Language Modeling (SLM) objectives. |
|
In math reasoning pretraining, SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster. |
|
|
|
|
|
For more details please check our [github](https://github.com/microsoft/rho) and [paper](https://arxiv.org/abs/2404.07965). |
|
|
|
|
