Model where??? :(((
Bad internet + bad pc setup = my life. Come back tomorrow, where hopefully the upload doesn't cut off before finishing, lol.
The suspense is killing me!!! We need to get you a better PC.
"the sheer, all-consuming, undeniable, unbreakable, unstoppable, irrevocable, eternal, everlasting, forever and always, now and forever, yes and amen,"
This model is interesting, it loves to use lots of adjectives
Edit: Just to tack on, it has a rare tendency to leak "```" at the end of messages
"the sheer, all-consuming, undeniable, unbreakable, unstoppable, irrevocable, eternal, everlasting, forever and always, now and forever, yes and amen,"
This model is interesting, it loves to use lots of adjectives
Edit: Just to tack on, it has a rare tendency to leak "```" at the end of messages
Well that's strange. I haven't seen this model do either of those behaviors before. Is this something that happens at high context? The only other thing I can think of is maybe the Nemo format or settings is slightly off, or maybe quant related. it's really weird to get format right.
This is the setup I use, plus the settings on the model card if that helps.
Well that's strange. I haven't seen this model do either of those behaviors before. Is this something that happens at high context? The only other thing I can think of is maybe the Nemo format or settings is slightly off, or maybe quant related. it's really weird to get format right.
This is the setup I use, plus the settings on the model card if that helps.
I do run Q4_K_M GGUF, mradermacher's non i-mat cause they were up at the time. It started happening more around 12Kish context.
Does disabling adding character names cause any issues? and are the "", "Example Separator" and "Chat Start" things recommended?
I use the default ST 1.12.4 included presets for context and instruct (Mistral), along with the recommended samplers.
I understand samplers, but instruct and context are beyond me x.x
Well that's strange. I haven't seen this model do either of those behaviors before. Is this something that happens at high context? The only other thing I can think of is maybe the Nemo format or settings is slightly off, or maybe quant related. it's really weird to get format right.
This is the setup I use, plus the settings on the model card if that helps.
I do run Q4_K_M GGUF, mradermacher's non i-mat cause they were up at the time. It started happening more around 12Kish context.
Does disabling adding character names cause any issues? and are the "", "Example Separator" and "Chat Start" things recommended?
I use the default ST 1.12.4 included presets for context and instruct (Mistral), along with the recommended samplers.
I understand samplers, but instruct and context are beyond me x.x
I'm still testing the models at the moment, mainly waiting until I was finished enhancing a dataset for better quants before doing any long-form chats at high context, but I feel like 12k is still kinda short for it to breakdown already. I haven't used GGUF too much though to get into specifics there. There is a dpo version of this merge which is similar, so you could try that as an alternative. See if that fixes it, if the slight advice below doesn't.
https://huggingface.co/mradermacher/Magnum-Instruct-DPO-12B-GGUF
I don't know what "Always add character's names to prompt" does specifically, but I've heard from others that some Nemo models perform better with it off, for some reason. It is different from "include Names" at least, since that's the one which actually displays which messages has what person/bot talking in the chat to help avoid confusion.
The example separator and chat start things can be whatever you what honestly. Only reason I have them like that is because my system prompt and cards now use triple hashes to denote different sections. One thing you might of missed is both the [INST] tokens need a space in front of them in the wrapping section, and no spaces behind, so that the chat logs look like this:
</s>[INST] USER: Hey, how are you?[/INST] BOT: I'm doing good. You?</s>[INST]
If you don't then the names will get unaligned in the chat, kinda like this:
</s>[INST]USER: Hey, how are you? [/INST]BOT: I'm doing good. You? </s>[INST]
Hopefully these tips solve it for you.
The example separator and chat start things can be whatever you what honestly. Only reason I have them like that is because my system prompt and cards now use triple hashes to denote different sections. One thing you might of missed is both the [INST] tokens need a space in front of them in the wrapping section, and no spaces behind, so that the chat logs look like this:
</s>[INST] USER: Hey, how are you?[/INST] BOT: I'm doing good. You?</s>[INST]If you don't then the names will get unaligned in the chat, kinda like this:
</s>[INST]USER: Hey, how are you? [/INST]BOT: I'm doing good. You? </s>[INST]Hopefully these tips solve it for you.
Prefix had (no space)INST
Suffix had (space)[/INST](no space)
I think i want?
Prefix (no space)INST
Suffix (no space)/INST
The example separator and chat start things can be whatever you what honestly. Only reason I have them like that is because my system prompt and cards now use triple hashes to denote different sections. One thing you might of missed is both the [INST] tokens need a space in front of them in the wrapping section, and no spaces behind, so that the chat logs look like this:
</s>[INST] USER: Hey, how are you?[/INST] BOT: I'm doing good. You?</s>[INST]If you don't then the names will get unaligned in the chat, kinda like this:
</s>[INST]USER: Hey, how are you? [/INST]BOT: I'm doing good. You? </s>[INST]Hopefully these tips solve it for you.
Prefix had (no space)INST
Suffix had (space)[/INST](no space)
I think i want?
Prefix (no space)INST
Suffix (no space)/INST
Looks like that, yeah. ST I think also defaults to using a newline in the assistant section, so delete that if you haven't already.
Supposedly it's best for Nemo models to have the conversation almost look like one continuous string, with no newlines to distinguish different roles, and only the actual chat/system messages themselves should ever have newlines. If you take a look at something like the Vicuna prompt, it has newlines between the User and Assistant section.
USER: {prompt}
ASSISTANT: {output}
That's something you want to avoid I guess with Nemo. You can check if you've done it right by doing a swipe/new message with the bot and checking Prompt Itemization's raw prompt logs inside SillyTavern, which is on the same row as the message edit button.