stablelm-2-zephyr-1_6b / tokenizer_config.json

Commit History

fix(tokenizer): set `mode_max_length=4096`
e1520e0
verified

jon-tow commited on

tmpfix(tokenizer_config): force `GPT2TokenizerFast`
542dce2
verified

jon-tow commited on

update(tokenizer): convert to `GPT2Tokenizer` (#15)
c89d7d1
verified

jon-tow commited on

Upload tokenizer
909ab34
verified

dmayhem93 commited on