Llama-31-8B_task-2_120-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2 and the GaetanMichelet/chat-120_ft_task-2 datasets. It achieves the following results on the evaluation set:

• Loss: 0.7144

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
1.1544	0.9091	5	1.1237
1.132	2.0	11	1.1193
1.0689	2.9091	16	1.1136
1.0956	4.0	22	1.1011
1.1157	4.9091	27	1.0871
1.0778	6.0	33	1.0639
1.0458	6.9091	38	1.0393
0.9854	8.0	44	1.0027
0.9996	8.9091	49	0.9696
0.8991	10.0	55	0.9317
0.8897	10.9091	60	0.9052
0.8711	12.0	66	0.8788
0.8809	12.9091	71	0.8588
0.7972	14.0	77	0.8368
0.8156	14.9091	82	0.8208
0.7815	16.0	88	0.8057
0.7492	16.9091	93	0.7956
0.7587	18.0	99	0.7855
0.7483	18.9091	104	0.7780
0.7296	20.0	110	0.7695
0.7441	20.9091	115	0.7629
0.7176	22.0	121	0.7561
0.7033	22.9091	126	0.7508
0.6906	24.0	132	0.7443
0.6954	24.9091	137	0.7396
0.6578	26.0	143	0.7344
0.6495	26.9091	148	0.7310
0.6391	28.0	154	0.7269
0.6442	28.9091	159	0.7237
0.6268	30.0	165	0.7199
0.6536	30.9091	170	0.7183
0.6092	32.0	176	0.7163
0.621	32.9091	181	0.7149
0.5823	34.0	187	0.7144
0.5651	34.9091	192	0.7156
0.5951	36.0	198	0.7164
0.5637	36.9091	203	0.7195
0.5669	38.0	209	0.7219
0.5613	38.9091	214	0.7278
0.5156	40.0	220	0.7309
0.5044	40.9091	225	0.7395

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1