metadata

license: apache-2.0
library_name: peft
tags:
  - trl
  - dpo
  - generated_from_trainer
base_model: TheBloke/OpenHermes-2-Mistral-7B-GPTQ
model-index:
  - name: openhermes-mistral-dpo-gptq
    results: []

openhermes-mistral-dpo-gptq

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

Loss: 0.6780
Rewards/chosen: 0.0314
Rewards/rejected: -0.0026
Rewards/accuracies: 0.6875
Rewards/margins: 0.0340
Logps/rejected: -157.6995
Logps/chosen: -203.0707
Logits/rejected: -2.3164
Logits/chosen: -2.4209

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 1
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 2
training_steps: 50
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6932	0.01	10	0.6910	-0.0090	-0.0070	0.25	-0.0020	-157.7437	-203.4745	-2.3168	-2.4207
0.6905	0.01	20	0.6893	-0.0028	-0.0049	0.1875	0.0022	-157.7232	-203.4128	-2.3159	-2.4207
0.6802	0.01	30	0.6868	0.0121	-0.0011	0.6875	0.0132	-157.6847	-203.2636	-2.3163	-2.4206
0.6872	0.02	40	0.6790	0.0245	-0.0033	0.6875	0.0278	-157.7068	-203.1401	-2.3163	-2.4207
0.7024	0.03	50	0.6780	0.0314	-0.0026	0.6875	0.0340	-157.6995	-203.0707	-2.3164	-2.4209

Framework versions

PEFT 0.9.0
Transformers 4.38.2
Pytorch 2.0.1+cu117
Datasets 2.18.0
Tokenizers 0.15.2