Low quants don't seem to work (no <reflection> tags)
I tried IQ2_M and IQ3_XS and none of them manage to output the or any other tag. They still display one first part made out of reasoning and a second part for the output, but the tags are missing.
This happens both using sillytavern (show tags is on) and Ooba as a front end.
(instruct prompt is proper)
Anyone else getting the same problem?
Yes, I'm having a similar issue even with the Q5_K_M quant. I can't get it to output any of the special tags likethinking or reflection. I'm using llama.cpp.
I use llama.cpp and see these tags with the option --special. From the documentation:
-sp, --special special tokens output enabled (default: false)
But i use 5 K L quantisation.
It's likely exactly as @supportend mentioned, just like you don't see the tags for chatting because they're special tags, they get hidden from rendering by default, but they're still there and the model still sees them and uses them
Is there a way to see the thinking or reflection tags when using Chat-Instruct mode in Oobabooga?
Yes there should also be a "special token" flag in oobabooga, it's on the parameters tab called "Skip special tokens", you'd need it to be off I think
Yes there should also be a "special token" flag in oobabooga, it's on the parameters tab called "Skip special tokens", you'd need it to be off I think
I'm not seeing a "Skip special tokens" in the parameters tab. I know it's supposed to be there as per the documentation: https://github.com/oobabooga/text-generation-webui/wiki/03-%E2%80%90-Parameters-Tab
The only thing I see is "Ban the eos_token -- Forces the model to never end the generation prematurely."
