Llama 3 coping mechanisms - Part 2
🤗
The same coping mechanisms but now in a porpose made space.
This is a direct Part 2 continuation of a discussion lost to the aether. Watching you cook gives me hopium.
#WaitingForLlama4
Llama 3 just released. Gaben has a Llama. Half-life 3 confirmed
DevsDoCode/LLama-3-8b-Uncensored
Haven't tried it yet due to the lack of gguf but I guess it's a step in the right direction?
Edit - Wrong formatting
@saishf But no model weights? :'(
:monkaToS:
Needs to be smooshed, I can try to do it. Put peft is scary
I've merged the Lora into the base model using mergekit but it seems rather lobotomized, you ask it how to make illicit substances and it explains why they are bad for you
My next try is going to be merging it with the instruct base instead
Edit - Bad spelling
it seems rather lobotomized, you ask it how to make illicit substances and it explains why they are bad for you
Useless like that.
it seems rather lobotomized, you ask it how to make illicit substances and it explains why they are bad for you
Useless like that.
Welp
You can fill in the blanks, don't know what the rules are for posted images
I did a few quick tests at 16K context. Llama 3 8B instruct loaded, but it seemed hallucinatory when probed about early context. Meta didn't reveal in config.json the maximum training context length, but if it was only 8K then I would not be surprised if the model needs a lot more training to get it to pay attention to larger context.
I found the paper for SOLAR, which documents what they used for training data sets, as well as their frankenmerge methodology using the base model, plus additional training to repair and tune it. Not that much data went into alignment compared to instruction. They doubled up the middle layers, effectively.
https://arxiv.org/abs/2312.15166
Extrapolating, I'd estimate that a Llama 3 SOLAR should clock in around 11.6B.
12B - the new hotness 😍
Working on a completely uncensored Aura model using Unholy as the base. I will not be testing zero shot because I personally don't care about zero shot performance. I will test thoroughly before uploading.
I have encountered an issue with quote/asterisk formatting, but plaintext/asterisk works fine. Attempting a fix with a LoRA that I have, but it is not quote/asterisk, so the problem may persist. Will let you know when I have it figured out.
I'm probably the only one that complains so much about quotes/asterisk formatting but I mean it from a place of heart.
@Lewdiculous Don't worry buddy I've got you in mind. I finally fixed my RP Format dataset for real. I just need to train it and then apply the lora. Once I get it trained it will be a band aid for any model that struggles with the format, hence the entire reason I do loras instead of fft in the first place.
LoRA inbound
It's like an optional DLC for now.
This is a direct continuation of this Part 2. Discussion Part 3 here.