tokenizer.model
#6
by
nds90
- opened
Hi! please post tokenizer.model
We are working on this
yes very much please do this and allow exl2's to be freely generated
is there any news on this tokenizer.model
?
Please - We really need the tokenizer.model
Thank you for your interest in DeepSeek Coder. Currently we can not provide a sentencepiece model (tokenizer.model) that achieves the same output as the existing HuggingFace Tokenizer. We are working hard to help the model quantization community directly support Huggingface's Tokenizer, and updates on our progress will be timely synced to https://github.com/deepseek-ai/DeepSeek-Coder#7-qa.
thanks for the update. Hopefully you take all the interest in the spirit it is given, as support for a fantastic achievement.
Chester111
changed discussion status to
closed