File size: 865 Bytes
8ddf596 97f00ee 8ddf596 97f00ee 8ddf596 d290234 8ddf596 cb51eb9 97f00ee |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
---
library_name: transformers
license: apache-2.0
datasets:
- bigscience-data/roots_zh-tw_wikipedia
- bigscience-data/roots_en_wikipedia
language:
- zh
---
# Model Card for Chinese-OpenELM-270M
Finetuned from [apple/OpenELM-270M](https://huggingface.co/apple/OpenELM-270M):
* Extended vocabulary from 32000 to 75873 with sentencepiece bpe trained on [bigscience-data/roots_zh-tw_wikipedia](https://huggingface.co/datasets/bigscience-data/roots_zh-tw_wikipedia) and used average embedding to initialize the new embeddings.
* Continual pre-trained with a mix of [bigscience-data/roots_zh-tw_wikipedia](https://huggingface.co/datasets/bigscience-data/roots_zh-tw_wikipedia) and [bigscience-data/roots_en_wikipedia](https://huggingface.co/datasets/bigscience-data/roots_en_wikipedia).
* Evaluation ppl = 1.6644828403646825 (split 3% training data as evaluation set) |