Llama-31-8B_task-1_180-samples_config-4_full

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-1, the GaetanMichelet/chat-120_ft_task-1 and the GaetanMichelet/chat-180_ft_task-1 datasets. It achieves the following results on the evaluation set:

• Loss: 0.9048

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
2.4745	0.9412	8	2.4335
2.4286	2.0	17	2.4114
2.419	2.9412	25	2.3814
2.3475	4.0	34	2.3262
2.3147	4.9412	42	2.2541
2.2214	6.0	51	2.1716
2.1097	6.9412	59	2.0745
1.9617	8.0	68	1.9479
1.908	8.9412	76	1.8375
1.7669	10.0	85	1.6953
1.6325	10.9412	93	1.5461
1.3201	12.0	102	1.3739
1.2477	12.9412	110	1.2331
1.163	14.0	119	1.1330
1.0579	14.9412	127	1.0861
1.0655	16.0	136	1.0611
0.9976	16.9412	144	1.0455
1.0285	18.0	153	1.0318
0.998	18.9412	161	1.0205
1.0038	20.0	170	1.0102
0.9907	20.9412	178	1.0020
0.9673	22.0	187	0.9929
0.95	22.9412	195	0.9870
0.9467	24.0	204	0.9801
0.9423	24.9412	212	0.9737
0.937	26.0	221	0.9675
0.9035	26.9412	229	0.9626
0.9074	28.0	238	0.9582
0.8944	28.9412	246	0.9534
0.8785	30.0	255	0.9493
0.8797	30.9412	263	0.9451
0.8764	32.0	272	0.9422
0.8903	32.9412	280	0.9389
0.8835	34.0	289	0.9377
0.8452	34.9412	297	0.9332
0.8777	36.0	306	0.9272
0.8101	36.9412	314	0.9257
0.8526	38.0	323	0.9229
0.8228	38.9412	331	0.9197
0.8066	40.0	340	0.9176
0.7701	40.9412	348	0.9199
0.8132	42.0	357	0.9162
0.7804	42.9412	365	0.9104
0.7508	44.0	374	0.9083
0.7192	44.9412	382	0.9052
0.7633	46.0	391	0.9048
0.7534	46.9412	399	0.9052
0.666	48.0	408	0.9151
0.7298	48.9412	416	0.9143
0.6815	50.0	425	0.9157
0.6845	50.9412	433	0.9170
0.6524	52.0	442	0.9216
0.6397	52.9412	450	0.9228

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1