phi-3-mini-QLoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6126

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 0
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 5
• training_steps: 250

Training results

Training Loss	Epoch	Step	Validation Loss
1.3693	0.6667	5	1.3378
1.1643	1.3333	10	1.1047
0.8388	2.0	15	0.8767
0.6894	2.6667	20	0.6828
0.5636	3.3333	25	0.5688
0.4496	4.0	30	0.5110
0.3487	4.6667	35	0.4549
0.3169	5.3333	40	0.4148
0.2595	6.0	45	0.3893
0.2002	6.6667	50	0.3733
0.2437	7.3333	55	0.3597
0.1669	8.0	60	0.3456
0.1873	8.6667	65	0.3491
0.1831	9.3333	70	0.3422
0.1581	10.0	75	0.3664
0.0831	10.6667	80	0.3644
0.1277	11.3333	85	0.3822
0.0539	12.0	90	0.3868
0.0799	12.6667	95	0.4190
0.066	13.3333	100	0.4375
0.0564	14.0	105	0.4581
0.0356	14.6667	110	0.4715
0.0493	15.3333	115	0.4896
0.0399	16.0	120	0.5066
0.0452	16.6667	125	0.5022
0.0305	17.3333	130	0.5246
0.036	18.0	135	0.5492
0.0282	18.6667	140	0.5537
0.0327	19.3333	145	0.5703
0.0341	20.0	150	0.5699
0.0315	20.6667	155	0.5761
0.0284	21.3333	160	0.5781
0.027	22.0	165	0.5818
0.0258	22.6667	170	0.5858
0.0224	23.3333	175	0.5884
0.0253	24.0	180	0.5960
0.0232	24.6667	185	0.6015
0.0256	25.3333	190	0.6088
0.0226	26.0	195	0.6106
0.0226	26.6667	200	0.6096
0.0259	27.3333	205	0.6102
0.0217	28.0	210	0.6100
0.022	28.6667	215	0.6115
0.0219	29.3333	220	0.6115
0.0239	30.0	225	0.6109
0.0226	30.6667	230	0.6123
0.0219	31.3333	235	0.6140
0.0201	32.0	240	0.6128
0.0198	32.6667	245	0.6130
0.0234	33.3333	250	0.6126

Framework versions

• PEFT 0.13.2
• Transformers 4.45.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0