Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /moss

2 contributors

History: 3 commits

xu-song's picture

add more tokenizers

f4973d4 12 months ago

moss-moon-003-sft
update about 1 year ago
20B_tokenizer.txt

6.82 MB

update about 1 year ago
README.md

570 Bytes

add more tokenizers 12 months ago
__init__.py

421 Bytes

add more tokenizers 12 months ago
convert_vocab_to_txt.py

615 Bytes

update about 1 year ago
test_arena.py

125 Bytes

add more tokenizers 12 months ago
test_decode.py

4.54 kB

update about 1 year ago
test_tokenizer.py

1.3 kB

add more tokenizers 12 months ago
test_zh_coding_len.py

2.38 kB

update about 1 year ago
vocab.zh.txt

1.54 MB

update about 1 year ago