Commit History

fix: make `eos_token`/`pad_token` overridable and add `pickle` support
4c846d7

jon-tow commited on

update(README): add extra lang tags
720763e
verified

jon-tow commited on

fix: add default special tokens
9f2def9
verified

jon-tow commited on

fix: add default special tokens
ca32832
verified

jon-tow commited on

fix: revert zephyr tokenizer config change
192936b
verified

jon-tow commited on

fix: add chat template to tokenizer config
507254d
verified

reciprocate commited on

feat: add `LICENSE`
e6185e1

jon-tow commited on

feat(tokenizer): expose merge ranks and special tokens for GGUF
a8f2f28
verified

jon-tow commited on

init: release
3aeae29

jon-tow commited on

initial commit
a3cceb4
verified

jon-tow commited on