metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
- GaetanMichelet/chat-60_ft_task-3
library_name: peft
license: llama3.1
tags:
- alignment-handbook
- trl
- sft
- generated_from_trainer
model-index:
- name: Llama-31-8B_task-3_60-samples_config-4
results: []
Llama-31-8B_task-3_60-samples_config-4
This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-3 dataset. It achieves the following results on the evaluation set:
- Loss: 0.4733
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 16
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 150
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.4485 | 0.6957 | 2 | 2.4786 |
| 2.4413 | 1.7391 | 5 | 2.4667 |
| 2.6263 | 2.7826 | 8 | 2.4443 |
| 2.1438 | 3.8261 | 11 | 2.4112 |
| 2.3995 | 4.8696 | 14 | 2.3654 |
| 2.2475 | 5.9130 | 17 | 2.3021 |
| 2.2194 | 6.9565 | 20 | 2.2092 |
| 2.2976 | 8.0 | 23 | 2.0988 |
| 2.0386 | 8.6957 | 25 | 2.0090 |
| 1.8757 | 9.7391 | 28 | 1.8478 |
| 1.753 | 10.7826 | 31 | 1.6617 |
| 1.5394 | 11.8261 | 34 | 1.4736 |
| 1.4055 | 12.8696 | 37 | 1.2968 |
| 1.1544 | 13.9130 | 40 | 1.1374 |
| 1.0965 | 14.9565 | 43 | 0.9952 |
| 0.8824 | 16.0 | 46 | 0.8632 |
| 0.8412 | 16.6957 | 48 | 0.7849 |
| 0.7232 | 17.7391 | 51 | 0.7002 |
| 0.6275 | 18.7826 | 54 | 0.6459 |
| 0.6078 | 19.8261 | 57 | 0.6102 |
| 0.5477 | 20.8696 | 60 | 0.5828 |
| 0.4369 | 21.9130 | 63 | 0.5587 |
| 0.5025 | 22.9565 | 66 | 0.5396 |
| 0.5043 | 24.0 | 69 | 0.5226 |
| 0.3742 | 24.6957 | 71 | 0.5101 |
| 0.449 | 25.7391 | 74 | 0.5003 |
| 0.3276 | 26.7826 | 77 | 0.4925 |
| 0.4754 | 27.8261 | 80 | 0.4932 |
| 0.3724 | 28.8696 | 83 | 0.4876 |
| 0.4679 | 29.9130 | 86 | 0.4861 |
| 0.3245 | 30.9565 | 89 | 0.4884 |
| 0.3613 | 32.0 | 92 | 0.4922 |
| 0.3511 | 32.6957 | 94 | 0.4899 |
| 0.5275 | 33.7391 | 97 | 0.4931 |
| 0.3403 | 34.7826 | 100 | 0.4883 |
| 0.4209 | 35.8261 | 103 | 0.4815 |
| 0.3543 | 36.8696 | 106 | 0.4805 |
| 0.4115 | 37.9130 | 109 | 0.4767 |
| 0.3902 | 38.9565 | 112 | 0.4794 |
| 0.3735 | 40.0 | 115 | 0.4776 |
| 0.3227 | 40.6957 | 117 | 0.4733 |
| 0.2983 | 41.7391 | 120 | 0.4797 |
| 0.4421 | 42.7826 | 123 | 0.4791 |
| 0.3819 | 43.8261 | 126 | 0.4739 |
| 0.2965 | 44.8696 | 129 | 0.4764 |
| 0.2661 | 45.9130 | 132 | 0.4765 |
| 0.3827 | 46.9565 | 135 | 0.4778 |
| 0.3144 | 48.0 | 138 | 0.4797 |
Framework versions
- PEFT 0.12.0
- Transformers 4.44.0
- Pytorch 2.1.2+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1