llama_finetune_arc_20

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.8405

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.4833	1.0	150	1.2465
0.7985	2.0	300	1.3109
0.7027	3.0	450	1.5734
0.2639	4.0	600	1.8512
0.1926	5.0	750	1.8805
0.1698	6.0	900	2.0632
0.1776	7.0	1050	2.1963
0.1594	8.0	1200	2.2222
0.1209	9.0	1350	2.1946
0.1142	10.0	1500	2.3561
0.1039	11.0	1650	2.4107
0.1015	12.0	1800	2.5198
0.1209	13.0	1950	2.4374
0.1324	14.0	2100	2.5536
0.1252	15.0	2250	2.6239
0.0989	16.0	2400	2.6389
0.139	17.0	2550	2.7198
0.1077	18.0	2700	2.7891
0.084	19.0	2850	2.8138
0.0765	20.0	3000	2.8405

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.0+cu121
• Datasets 2.13.1
• Tokenizers 0.14.1