Update README.md
Browse files
README.md
CHANGED
|
@@ -15,7 +15,7 @@ pipeline_tag: text-generation
|
|
| 15 |
|
| 16 |
[paper](https://arxiv.org/abs/2410.17215) | [code](https://github.com/thu-coai/MiniPLM)
|
| 17 |
|
| 18 |
-
**Pretrain-Qwen-500M** is a 500M model with
|
| 19 |
|
| 20 |
We also open-source the tokenized [pre-training corpus](https://huggingface.co/datasets/MiniLLM/pile-tokenized) for reproducibility.
|
| 21 |
|
|
|
|
| 15 |
|
| 16 |
[paper](https://arxiv.org/abs/2410.17215) | [code](https://github.com/thu-coai/MiniPLM)
|
| 17 |
|
| 18 |
+
**Pretrain-Qwen-500M** is a 500M model with Qwen achitecture conventionally pre-trained from scratch on [the Pile](https://huggingface.co/datasets/monology/pile-uncopyrighted) for 50B tokens.
|
| 19 |
|
| 20 |
We also open-source the tokenized [pre-training corpus](https://huggingface.co/datasets/MiniLLM/pile-tokenized) for reproducibility.
|
| 21 |
|