Mistral-7B-Instruct-v0.3-ORPO-SALT

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the dpo_mix_en and the bct_non_cot_dpo_1000 datasets. It achieves the following results on the evaluation set:

• Loss: 0.8434
• Rewards/chosen: -0.0777
• Rewards/rejected: -0.0988
• Rewards/accuracies: 0.5691
• Rewards/margins: 0.0210
• Logps/rejected: -0.9877
• Logps/chosen: -0.7773
• Logits/rejected: -3.1073
• Logits/chosen: -3.0834
• Sft Loss: 0.7773
• Odds Ratio Loss: 0.6614

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen	Sft Loss	Odds Ratio Loss
0.8803	0.8082	500	0.8619	-0.0796	-0.0983	0.5655	0.0187	-0.9834	-0.7962	-3.0746	-3.0520	0.7962	0.6572
0.7341	1.6165	1000	0.8450	-0.0779	-0.0980	0.5673	0.0201	-0.9804	-0.7795	-3.1194	-3.0960	0.7795	0.6550
0.7691	2.4247	1500	0.8434	-0.0777	-0.0988	0.5691	0.0210	-0.9877	-0.7773	-3.1073	-3.0834	0.7773	0.6614

Framework versions

• PEFT 0.10.0
• Transformers 4.40.1
• Pytorch 2.3.0
• Datasets 2.19.0
• Tokenizers 0.19.1