lmind_nq_train6000_eval6489_v1_docidx_v3_Qwen_Qwen1.5-4B_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3 dataset. It achieves the following results on the evaluation set:

• Loss: 5.3392
• Accuracy: 0.4286

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.9569	0.9985	341	0.4736	3.0300
1.8799	2.0	683	0.468	3.0993
1.7649	2.9985	1024	0.4650	3.2750
1.6077	4.0	1366	0.4625	3.4406
1.4321	4.9985	1707	0.4586	3.6500
1.2382	6.0	2049	0.4562	3.8598
1.0525	6.9985	2390	0.4541	4.0638
0.8607	8.0	2732	0.4515	4.2389
0.7099	8.9985	3073	0.4516	4.3484
0.5823	9.9854	3410	0.4488	4.5794
0.4641	10.9985	3751	4.7090	0.4495
0.3755	12.0	4093	4.9454	0.4354
0.3235	12.9985	4434	5.0624	0.4379
0.2691	14.0	4776	5.0957	0.4345
0.2394	14.9985	5117	5.1831	0.4368
0.2112	16.0	5459	5.3223	0.4326
0.1994	16.9985	5800	5.3839	0.4301
0.1834	18.0	6142	5.4236	0.4286
0.1709	18.9985	6483	5.4840	0.4291
0.166	19.9854	6820	5.3392	0.4286

Framework versions

• PEFT 0.5.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1