Hugging Face's logo

Spaces:
xu-song
/
tokenizer-arena
Running

App Files Community
1
tokenizer-arena / util.py

Commit History

add compress rate
814ee6b

xu-song commited on

fix tiktoken
a6c67ec

xu-song commited on

add more tokenizer
5425d5d

xu-song commited on

fix tokenize
e6543ac

xu-song commited on

fix chatglm; new feature about add_special_tokens;
d27a756

xu-song commited on

update
a37f943

xu-song commited on

update
ae282a4

xu-song commited on

add more tokenizers
f4973d4

xu-song commited on

update
7cb27ea

xu-song commited on

update
7a8d6d6

xu-song commited on

update
8ba122d

xu-song commited on

update
9495a4f

xu-song commited on

update
9dfa838

xu-song commited on

update
6f9d07b

xu-song commited on

update
0177868

xu-song commited on

update
79b95c3

xu-song commited on

startup
b15345c

xu-song commited on

update
d10ecd7

xu-song commited on