|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
--- |
|
|
|
# llama-161M |
|
|
|
Trained on 100B tokens. |
|
- 1e-3 LR |
|
- 0.1 wd |
|
- WSD scheduler with 10% decay |
|
- 80% code, 10% NL, 10% instruction data |
|
- Dataset decontaminated against popular benchmarks following [bigcode](https://github.com/bigcode-project/bigcode-dataset/tree/main/decontamination) |
|
- 8x3090s 110~ hours |
|
|
|
|
|
This is a *base* pretrained model and requires further fine tuning to be useful. |
|
|
|
## Model Details |
|
|
|
| [openai/openai_humaneval](https://huggingface.co/datasets/openai/openai_humaneval) (greedy) | [mbpp](https://huggingface.co/datasets/google-research-datasets/mbpp) (greedy) | |
|
| :------------------ | :------------- | |
|
| 9.2% | 9.8% | |
