template problem,
Hi, I trying to implement chat "continue" using api with this model, so I found that sometime continue "not good" after short debug, i found that it is because api send </s> token after request, which breaks previous line, so may be I could ask you to update the template here, to allow enduser to provide additional payload with a message, to prevent closing token
like change this part with additional condition on if add_generation_prompt property as in some other templates
{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}
or may be you know where this template from, so we can ask to update him too :)
great model btw !
I'm encountering a common issue with the model ending with wrong variations of <|im_end|> since it's technically multiple tokens instead of one, so it has a higher chance of outputting wrong end of turn token(s).
<|im_end|>
Tokens: [523, 28766, 321, 28730, 416, 28766, 28767]
which is honestly terrible mistake since it sometimes outputs wrong tokens like: <|user|>, <||, <|\n<user>
Edit: I should have opened another issue instead.
@Bakanayatsu
model has another themplate - <s>[INST]</s> look into tokenizer.config
@2dts isn't the template ChatML as per the model card? I know this model is initialized from Mistral-7B-Instruct-v0.2 so that template remained in tokenizer.config.
Ok, for ppl who meet same problem - i just placed "not loop.last" condition to eos token, as there is no reason set </s> when there was last reply from bot ("continue" is only case when it happens)