ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V5

This model is a fine-tuned version of taide/TAIDE-LX-7B-Chat on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.3726

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
4.8629	0.0321	100	3.4698
4.6474	0.0643	200	3.2343
3.8261	0.0964	300	2.8205
3.2992	0.1285	400	2.5034
2.9369	0.1607	500	2.2639
2.2674	0.1928	600	1.9493
2.1393	0.2249	700	1.7888
1.8195	0.2571	800	1.6285
1.678	0.2892	900	1.4983
1.6242	0.3213	1000	1.4013
1.344	0.3535	1100	1.1522
1.0894	0.3856	1200	1.0704
1.2033	0.4177	1300	1.0537
0.9503	0.4499	1400	0.9160
0.9901	0.4820	1500	0.8751
1.0363	0.5141	1600	0.7942
0.9986	0.5463	1700	0.7668
0.9407	0.5784	1800	0.6912
0.9347	0.6105	1900	0.6543
0.8109	0.6427	2000	0.6498
0.8848	0.6748	2100	0.6077
0.8937	0.7069	2200	0.5865
0.7895	0.7391	2300	0.5780
0.8044	0.7712	2400	0.5646
0.8317	0.8033	2500	0.5449
0.858	0.8355	2600	0.5132
0.8519	0.8676	2700	0.4940
0.7554	0.8997	2800	0.4972
0.758	0.9319	2900	0.4809
0.8866	0.9640	3000	0.4714
0.7028	0.9961	3100	0.4608
0.7031	1.0283	3200	0.4458
0.6623	1.0604	3300	0.4427
0.671	1.0925	3400	0.4366
0.6588	1.1247	3500	0.4327
0.6422	1.1568	3600	0.4239
0.643	1.1889	3700	0.4235
0.6747	1.2211	3800	0.4204
0.6911	1.2532	3900	0.4130
0.7354	1.2853	4000	0.4092
0.6233	1.3175	4100	0.4070
0.6005	1.3496	4200	0.4055
0.624	1.3817	4300	0.4033
0.623	1.4139	4400	0.3976
0.6419	1.4460	4500	0.3966
0.6329	1.4781	4600	0.3914
0.6395	1.5103	4700	0.3934
0.6541	1.5424	4800	0.3916
0.6538	1.5746	4900	0.3917
0.6214	1.6067	5000	0.3840
0.6303	1.6388	5100	0.3844
0.6547	1.6710	5200	0.3816
0.6264	1.7031	5300	0.3844
0.5896	1.7352	5400	0.3801
0.6082	1.7674	5500	0.3786
0.5772	1.7995	5600	0.3737
0.5839	1.8316	5700	0.3745
0.6201	1.8638	5800	0.3737
0.5643	1.8959	5900	0.3717
0.6258	1.9280	6000	0.3726

Framework versions

• PEFT 0.12.1.dev0
• Transformers 4.44.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1