Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /gpt_35_turbo

2 contributors

History: 13 commits

xu-song's picture

add compress rate

814ee6b 7 months ago

README.md

1.65 kB

fix chatglm; new feature about add_special_tokens; 10 months ago
__init__.py

348 Bytes

add compress rate 7 months ago
cl100k_base.tiktoken

1.68 MB

update about 1 year ago
decode_test.py

492 Bytes

add compress rate 7 months ago
test_arena.py

199 Bytes

add more tokenizers 12 months ago
test_tiktoken.py

1.19 kB

add compress rate 7 months ago
tiktoken_decode_test.py

965 Bytes

add more tokenizers 12 months ago
vocab.jsonl

3.79 MB

add compress rate 7 months ago