library_name: transformers
license: apache-2.0
datasets:
- liswei/zhtw-news-and-articles-2B
base_model:
- liswei/Taiwan-ELM
- apple/OpenELM-270M
language:
- zh
metrics:
- perplexity
pipeline_tag: text-generation
Efficient LLM for Taiwan
Taiwan ELM
Taiwan ELM is a family of Efficient LLMs for Taiwan base on apple/OpenELM. The project aims to provide an efficient model for researchers without access to large-scale computing resources.
The model is trained using a custom fork of LLaMA-Factory on 2B Traditional Chinese tokens and 500K instruction samples. We will extend the model to train on larger data sets and different base models if there is sufficient demand.
What is being released?
We release both pre-trained base models and instruction tuned variants with 270M and 1.1B parameters. Along with the model, datasets used to train the base and instruction-tuned models are also released.
List of released models:
- Taiwan-ELM-270M
- Taiwan-ELM-1_1B
- Taiwan-ELM-270M-Instruct
- Taiwan-ELM-1_1B-Instruct
List of released datasets:
Usage Examples
We adapt the LLaMA2 template:
<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>
{{ user_message }} [/INST]
The model could be load via AutoModelForCausalLM with trust_remote_code=True:
taiwanelm_270m = AutoModelForCausalLM.from_pretrained("liswei/Taiwan-ELM-270M", trust_remote_code=True)
We also support additional generation methods and speculative generation, please find reference at OpenELM#usage.