README.md

metadata

title: Tokenizer Arena
emoji: ⚡
colorFrom: red
colorTo: gray
sdk: gradio
sdk_version: 4.31.4
app_file: app.py
pinned: false
datasets:
  - cc100

压缩率 Compress Rate

在 cc-100 数据集，每个语言取1万条数据，测试不同tokenizer的压缩率。

压缩率示例： llama3扩充了词典，具有更高的压缩比。同样1T字节的简体中文语料，llama分词后是 0.56万亿个token，llama3只需要0.31万亿个token。

tokenizer	vocab_size	t_bytes/t_tokens	t_tokens/t_bytes	n_chars/n_tokens
llama	32000	1.8	0.56	0.7
llama3	128000	3.2	0.31	1.24

可通过以下脚本进行复现

python utils/compress_rate_util.py 

英文压缩率

在英文数据集 cc100-en 计算压缩率

tokenizer	vocab_size	g_bytes/b_tokens	b_tokens/g_bytes	t_bytes/t_tokens	t_tokens/t_bytes	n_chars/n_tokens
amber	32000	3.56	0.28	3.47	0.29	3.81
aya_101	250100	3.3	0.3	3.22	0.31	3.53
baichuan	64000	3.74	0.27	3.65	0.27	4
baichuan2	125696	3.89	0.26	3.8	0.26	4.17

简体中文压缩率

在简体中文数据集 cc100-zh-Hans 计算压缩率

tokenizer	vocab_size	g_bytes/b_tokens	b_tokens/g_bytes	t_bytes/t_tokens	t_tokens/t_bytes	n_chars/n_tokens
amber	32000	1.84	0.54	1.8	0.56	0.7
aya_101	250100	3.89	0.26	3.79	0.26	1.47
baichuan	64000	3.92	0.26	3.82	0.26	1.48

Reference

• Getting the most out of your tokenizer for pre-training and domain adaptation
• Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

• blog - https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them - https://huggingface.co/docs/transformers/tokenizer_summary#sentencepiece - https://www.huaxiaozhuan.com/%E5%B7%A5%E5%85%B7/huggingface_transformer/chapters/1_tokenizer.html - https://zhuanlan.zhihu.com/p/652520262 - https://github.com/QwenLM/Qwen/blob/main/tokenization_note_zh.md - https://tonybaloney.github.io/posts/cjk-chinese-japanese-korean-llm-ai-best-practices.html

• demo
  • https://huggingface.co/spaces/Xenova/the-tokenizer-playground
  • https://github.com/dqbd/tiktokenizer
  • https://chat.lmsys.org/?leaderboard
  • https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
• paper
  • ss