stablelm-2-1_6b / tokenization_arcade100k.py

Commit History

fix(tokenizer): expose `errors`
21ee10d
verified

jon-tow commited on

fix: make `eos_token`/`pad_token` overridable and add `pickle` support
4c846d7

jon-tow commited on

feat(tokenizer): expose merge ranks and special tokens for GGUF
a8f2f28
verified

jon-tow commited on

init: release
3aeae29

jon-tow commited on