Update README.md

README.md

@@ -3,7 +3,9 @@ library_name: transformers
 license: apache-2.0
 datasets:
 - liswei/zhtw-news-and-articles-2B
-base_model: apple/OpenELM-270M
+base_model: 
+- apple/OpenELM-270M
+- liswei/Taiwan-ELM
 language:
 - zh
 metrics:
@@ -11,16 +13,49 @@ metrics:
 pipeline_tag: text-generation
 ---
 
-# Model Card for Chinese-OpenELM-270M
+<center>
+    <img src="https://huggingface.co/liswei/Taiwan-ELM/resolve/main/Taiwan%20ELM%20Logo.jpeg" alt="Efficient LLM for Taiwan">
+</center>
 
-Continual pre-trained from [apple/OpenELM-270M](https://huggingface.co/apple/OpenELM-270M) with [liswei/zhtw-news-and-articles-2B](https://huggingface.co/datasets/liswei/zhtw-news-and-articles-2B):
+> Efficient LLM for Taiwan
 
-* Extended vocabulary from 32000 to 61758 tokens with additional Traditional Chinese characters.
-  * Tokenizer is trained on [liswei/zhtw-news-and-articles-2B](https://huggingface.co/datasets/liswei/zhtw-news-and-articles-2B) and pruned from 96000 to 61758 tokens while maintaining 95% coverage on the pre-training dataset.
-  * Additional token embeddings are initialized with the mean vector of existing embeddings.
-* Traditional Chinese perplexity = 1.6871 on held-out evaluation dataset.
-* Applied [GaLore](https://arxiv.org/abs/2403.03507) for efficient training with following hyperparameters:
-  * Rank: 1024
-  * Scale: 4.0
-  * Update interval: 200
-  * Layer-wise training: False
+# Taiwan ELM
+
+Taiwan ELM is a family of Efficient LLMs for Taiwan base on [apple/OpenELM](https://huggingface.co/apple/OpenELM).
+The project aims to provide an efficient model for researchers without access to large-scale computing resources.
+
+The model is trained using a custom fork of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) on 2B Traditional Chinese tokens and 500K instruction samples.
+We will extend the model to train on larger data sets and different base models if there is sufficient demand.
+
+## What is being released?
+
+We release both pre-trained base models and instruction tuned variants with 270M and 1.1B parameters.
+Along with the model, datasets used to train the base and instruction-tuned models are also released.
+
+List of released models:
+* Taiwan-ELM-270M
+* Taiwan-ELM-1_1B
+* Taiwan-ELM-270M-Instruct
+* Taiwan-ELM-1_1B-Instruct
+
+List of released datasets:
+* [liswei/Taiwan-Text-Excellence-2B](https://huggingface.co/datasets/liswei/Taiwan-Text-Excellence-2B)
+* [liswei/PromptPair-TW](https://huggingface.co/datasets/liswei/PromptPair-TW)
+
+## Usage Examples
+
+We adapt the LLaMA2 template:
+```jinja2
+<s>[INST] <<SYS>>
+{{ system_prompt }}
+<</SYS>>
+
+{{ user_message }} [/INST]
+```
+
+The model could be load via `AutoModelForCausalLM` with `trust_remote_code=True`:
+```python
+taiwanelm_270m = AutoModelForCausalLM.from_pretrained("liswei/Taiwan-ELM-270M", trust_remote_code=True)
+```
+
+We also support additional generation methods and speculative generation, please find reference at [OpenELM#usage](https://huggingface.co/apple/OpenELM#usage).