Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-3_120-samples_config-4

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-3_120-samples_config-4 / README.md
GaetanMichelet's picture
GaetanMichelet
End of training
075b20a verified
preview code
|
raw
history blame contribute delete
4.12 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-3
  - GaetanMichelet/chat-120_ft_task-3
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-3_120-samples_config-4
    results: []

Llama-31-8B_task-3_120-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-3 and the GaetanMichelet/chat-120_ft_task-3 datasets. It achieves the following results on the evaluation set:

  • Loss: 0.4427

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
2.4707 0.9091 5 2.5024
2.3847 2.0 11 2.4794
2.5822 2.9091 16 2.4506
2.2635 4.0 22 2.3903
2.446 4.9091 27 2.3184
2.3172 6.0 33 2.1938
2.0582 6.9091 38 2.0560
1.9038 8.0 44 1.8385
1.7291 8.9091 49 1.6252
1.3996 10.0 55 1.3456
1.1127 10.9091 60 1.1189
0.8648 12.0 66 0.8627
0.8247 12.9091 71 0.7228
0.5681 14.0 77 0.6453
0.4968 14.9091 82 0.6020
0.589 16.0 88 0.5632
0.3902 16.9091 93 0.5394
0.4795 18.0 99 0.5253
0.3937 18.9091 104 0.5188
0.3482 20.0 110 0.5149
0.4633 20.9091 115 0.5076
0.4324 22.0 121 0.5073
0.5268 22.9091 126 0.5014
0.3829 24.0 132 0.4946
0.3884 24.9091 137 0.4875
0.3955 26.0 143 0.4859
0.3296 26.9091 148 0.4827
0.364 28.0 154 0.4805
0.3218 28.9091 159 0.4765
0.2995 30.0 165 0.4727
0.3728 30.9091 170 0.4668
0.2413 32.0 176 0.4653
0.4141 32.9091 181 0.4626
0.3693 34.0 187 0.4562
0.3666 34.9091 192 0.4526
0.3605 36.0 198 0.4506
0.2923 36.9091 203 0.4474
0.3422 38.0 209 0.4443
0.3489 38.9091 214 0.4427
0.3737 40.0 220 0.4435
0.2804 40.9091 225 0.4437
0.3212 42.0 231 0.4469
0.2322 42.9091 236 0.4480
0.2569 44.0 242 0.4507
0.2501 44.9091 247 0.4544
0.2247 46.0 253 0.4641

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1