pipeline_tag: text-generation
license: apache-2.0
language:
- zh
Model Card for Breeze-7B-Instruct-v0.1
Breeze-7B is a language model that builds upon the foundation of Mistral-7B-v0.1, specifically enhanced for Traditional Chinese.
Breeze-7B-Base-v0.1 introduces an expanded vocabulary with additional 30,000 Traditional Chinese tokens and is pre-trained on a substantial dataset of 250GB of Traditional Chinese content. With the expanded vocabulary, the base model operates at twice the inference speed for Traditional Chinese characters compared to Mistral-7B. [See Inference Performance.] This achievement marks a significant milestone as it is the first instance of vocabulary expansion in a model tailored for Traditional Chinese.
Breeze-7B-Instruct-v0.1 derives from the base model Breeze-7B-Base-v0.1 and has undergone supervised fine-tuning with over 1 million instances to sharpen its capabilities. This fine-tuned model demonstrates impressive performance in benchmarks for both English and Traditional Chinese, surpassing the results of Taiwan-LLM-7B-v2.1-chat, Taiwan-LLM-13B-v2.0-chat and Qwen-7B-chat in Traditional Chinese assessments. It also excels in some benchmarks against Yi-6B-Chat. In English evaluations, Breeze-7B-Instruct-v0.1 shows comparable results to Mistral-7B-Instruct-v0.1 on the MMLU and MT-Bench benchmarks. [See Chat Model Performance.]
Breeze-7B-Instruct-64k-v0.1 is an extension to Breeze-7B-Instruct-v0.1 to enable 64k context length, which is equivalent to 88k Traditional Chinese characters. With minimal sacrifice in the performance of the regular benchmarks, Breeze-7B-Instruct-64k-v0.1 can solve tasks such as question answering and summarization on document-level inputs. [See Long-context Performance.]
A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.
Features
- Breeze-7B-Base-v0.1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 8k tokens context length
- Breeze-7B-Instruct-v0.1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 8k tokens context length
- Multi-turn dialogue (without special handling for harmfulness)
- Breeze-7B-Instruct-64k-v0.1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 64k tokens context length
- Multi-turn dialogue (without special handling for harmfulness)
Model Details
- Breeze-7B-Base-v0.1
- Finetuned from: mistralai/Mistral-7B-v0.1
- Model type: Causal decoder-only transformer language model
- Language: English and Traditional Chinese (zh-tw)
- Breeze-7B-Instruct-v0.1
- Finetuned from: MediaTek-Research/Breeze-7B-Base-v0.1
- Model type: Causal decoder-only transformer language model
- Language: English and Traditional Chinese (zh-tw)
- Breeze-7B-Instruct-64k-v0.1
- Finetuned from: MediaTek-Research/Breeze-7B-Instruct-v0.1
- Model type: Causal decoder-only transformer language model
- Language: English and Traditional Chinese (zh-tw)
Base Model Performance
| Models | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC) | |
|---|---|---|---|---|---|
| TC, Knowledge | TC, Reasoning | TC, Reasoning | EN, Knowledge | ||
| 5 shot | 3 shot | 5 shot | 5 shot | ||
| Yi-34B | 34B | 63.10 | 84.57 | 49.31 | 77.42 |
| Qwen-14B | 14B | 51.30 | 16.95 * | 50.69 | 68.83 |
| Yi-6B | 6B | 49.63 | 76.61 | 34.72 | 65.35 |
| Qwen-7B | 7B | 42.84 | 0.0 * | 39.58 | 61.00 |
| Breeze-7B-Base-v0.1 | 7B | 40.35 | 81.13 | 28.47 | 61.63 |
| Mistral-7B-v0.1 | 7B | 36.93 | 79.27 | 27.78 | 64.89 |
* Few-shot learning cannot effectively guide the model to generate the proper answer.
| Category ACC of TMMLU+ (5 shot) | STEM | Social Science | Humanities | Other |
|---|---|---|---|---|
| Yi-34B | 56.03 | 73.06 | 61.12 | 62.19 |
| Qwen-14B | 46.51 | 58.20 | 51.12 | 49.38 |
| Yi-6B | 41.14 | 57.77 | 50.22 | 49.39 |
| Qwen-7B | 28.25 | 47.80 | 43.14 | 42.17 |
| Breeze-7B-Base-v0.1 | 35.74 | 46.08 | 40.29 | 39.27 |
| Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 |
TMMLU+, DRCD, and Table source from MediaTek-Research/TCEval-v2. MediaTek-Research/TCEval-v2 derives from TCEval-v1 and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train. We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, DRCD, Table, and MMLU.
Chat Model Performance
| Models | TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MT-Bench-tw (Score) | MMLU (ACC) | MMLU (ACC) | MT-Bench (Score) | |
|---|---|---|---|---|---|---|---|---|---|
| TC, Knowledge | TC, Knowledge | TC, Reasoning | TC, Reasoning | TC, Chat | EN, Knowledge | EN, Knowledge | EN, Chat | ||
| 0 shot | 5 shot | 3 shot | 0 shot | 0 shot | 0 shot | 5 shot | 0 shot | ||
| Yi-34B-Chat | 34B | 54.87 | 36.81 | 6.9 | 71.04 | 7.6 | |||
| Qwen-14B-Chat | 14B | 48.41 | 41.67 | 6.4 | 64.91 | 7.2 | |||
| Yi-6B-Chat | 6B | 44.79 | 25.69 | 5.0 | 59.45 | 6.0 | |||
| gpt-3.5-turbo | 41.76 | 7.1 | 70.00 | 7.9 | |||||
| Breeze-7B-Instruct-v0.1 | 7B | 41.61 | 45.83 | 5.7 | 63.26 | 7.1 | |||
| Breeze-7B-Instruct-64k-v0.1 | 7B | 40.99 | 36.11 | 5.5 | 63.68 | 7.1 | |||
| Qwen-7B-Chat | 7B | 40.02 | 33.33 | 5.4 | 55.94 | 6.2 | |||
| Taiwan-LLM-13B-v2.0-chat | 13B | 29.47 | 23.61 | 5.0 | 50.50 | -* | |||
| Taiwan-LLM-7B-v2.1-chat | 7B | 28.08 | 31.25 | 4.2 | 42.72 | -* |
* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
| Category ACC of TMMLU+ (0 shot) | STEM | Social Science | Humanities | Other |
|---|---|---|---|---|
| Yi-34B-Chat | 47.65 | 64.25 | 52.73 | 54.91 |
| Qwen-14B-Chat | 43.83 | 55.00 | 48.55 | 46.22 |
| Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 |
| gpt-3.5-turbo | 41.56 | 46.72 | 36.73 | 42.03 |
| Breeze-7B-Instruct-v0.1 | 37.41 | 46.81 | 42.06 | 40.16 |
| Breeze-7B-Instruct-64k-v0.1 | 37.88 | 46.35 | 40.31 | 39.40 |
| Qwen-7B-Chat | 35.44 | 46.22 | 38.35 | 40.06 |
| Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 |
| Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 |
TMMLU+, DRCD, Table, and MT-Bench-tw source from MediaTek-Research/TCEval-v2. MediaTek-Research/TCEval-v2 derives from TCEval-v1 and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train. MT-Bench source from lmsys/mt_bench_human_judgments. We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, DRCD, Table, and MMLU. We use the code revised from fastchat llm_judge to evaluate MT-Bench-tw and MT-Bench.
Inference Performance
In this test, we use the first 700 characters of the web article as the input and ask the model to write the same article again.
All inferences run on 2 RTX A6000 GPUs (using vllm, with a tensor-parallel size of 2).
| Models | Inference Time (sec) | Estimated Max Input Length (Char) |
|---|---|---|
| Yi-6B | 10.62 | 5.2k |
| Breeze-7B-Instruct-v0.1 | 10.74 | 11.1k |
| Breeze-7B-Instruct-64k-v0.1 | 10.74 | 88.8k |
| Qwen-7B | 10.86 | 9.8k |
| Qwen-14B | 18.89 | 9.8k |
| Mistral-7B-v0.1 | 20.48 | 5.1k |
| Taiwan-LLM-7B-v2.1-base | 26.26 | 2.2k |
| Taiwan-LLM-13B-v2.0-base | 36.80 | 2.2k |
| Yi-34B | 43.71 | 4.5k |
Long-context Performance
TBD
Examples
TBD
Use in Transformers
First install direct dependencies:
pip install transformers torch accelerate
If you want faster inference using flash-attention2, you need to install these dependencies:
pip install packaging ninja
pip install flash-attn
Then load the model in transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
model="MediaTek-Research/Breeze-7B-Instruct-v0.1",
device_map="auto",
torch_dtype=torch.bfloat16,
use_flash_attn_2=True # optional
)
The structure of the query template follows that of Mistral-7B-Instruct, as shown below.
<s> SYS_PROMPT [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST]
where SYS_PROMPT, QUERY1, RESPONSE1, and QUERY2 can be provided by the user.
The suggested default SYS_PROMPT is
You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.