Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /gpt_35_turbo

2 contributors

History: 7 commits

xu-song's picture

add more tokenizers

f4973d4 12 months ago

README.md

1.47 kB

add more tokenizers 12 months ago
__init__.py

2.17 kB

add more tokenizers 12 months ago
cl100k_base.tiktoken

1.68 MB

update about 1 year ago
decode_test.py

126 Bytes

add more tokenizers 12 months ago
test_arena.py

199 Bytes

add more tokenizers 12 months ago
test_tiktoken.py

1.03 kB

add more tokenizers 12 months ago
tiktoken_decode_test.py

965 Bytes

add more tokenizers 12 months ago
vocab.jsonl

3.78 MB

add more tokenizers 12 months ago