Hugging Face's logo

yiran-wang3
/
ds_chat_sppo_hard_new_iter0_constant_1e-6

Safetensors
llama
alignment-handbook
trl
dpo
Generated from Trainer
Model card Files Community
ds_chat_sppo_hard_new_iter0_constant_1e-6 / README.md
yiran-wang3's picture
yiran-wang3
End of training
26ff372 verified
preview code
|
raw
history blame contribute delete
No virus
3.57 kB
metadata
license: other
base_model: deepseek-ai/deepseek-llm-7b-chat
tags:
  - alignment-handbook
  - trl
  - dpo
  - generated_from_trainer
  - trl
  - dpo
  - generated_from_trainer
datasets:
  - self-generate/ds_chat_original_cn_mining_oj_iter0-binarized
  - self-generate/ds_chat_original_cn_rl_oj_iter0-binarized
model-index:
  - name: ds_chat_sppo_hard_new_iter0_2024-09-17-14.35
    results: []

Visualize in Weights & Biases

ds_chat_sppo_hard_new_iter0_2024-09-17-14.35

This model is a fine-tuned version of deepseek-ai/deepseek-llm-7b-chat on the self-generate/ds_chat_original_cn_mining_oj_iter0-binarized and the self-generate/ds_chat_original_cn_rl_oj_iter0-binarized datasets. It achieves the following results on the evaluation set:

  • Loss: 0.4365
  • Rewards/chosen: 0.0466
  • Rewards/rejected: -0.0388
  • Rewards/accuracies: 0.5769
  • Rewards/margins: 0.0854
  • Logps/rejected: -287.4509
  • Logps/chosen: -250.6894
  • Logits/rejected: 1.4344
  • Logits/chosen: 1.2252
  • Debug/policy Chosen Logits: 1.2252
  • Debug/policy Rejected Logits: 1.4344
  • Debug/policy Chosen Logps: -250.6894
  • Debug/policy Rejected Logps: -287.4509
  • Debug/reference Chosen Logps: -255.3497
  • Debug/reference Rejected Logps: -283.5705

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-06
  • train_batch_size: 8
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • total_train_batch_size: 64
  • total_eval_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.1
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 1.0

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen Debug/policy Chosen Logits Debug/policy Rejected Logits Debug/policy Chosen Logps Debug/policy Rejected Logps Debug/reference Chosen Logps Debug/reference Rejected Logps
0.4429 0.6329 100 0.4365 0.0466 -0.0388 0.5769 0.0854 -287.4509 -250.6894 1.4344 1.2252 1.2252 1.4344 -250.6894 -287.4509 -255.3497 -283.5705

Framework versions

  • Transformers 4.42.0
  • Pytorch 2.3.0+cu121
  • Datasets 2.14.6
  • Tokenizers 0.19.1