ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V2

This model is a fine-tuned version of taide/TAIDE-LX-7B-Chat on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4516

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.9094	0.0321	100	2.2941
2.1923	0.0643	200	1.7722
1.7338	0.0964	300	1.4302
1.4633	0.1285	400	1.2009
1.0927	0.1607	500	1.0950
1.0335	0.1928	600	0.9723
1.0519	0.2249	700	0.8645
0.8477	0.2571	800	0.8000
0.8899	0.2892	900	0.7260
0.9374	0.3213	1000	0.6832
0.9934	0.3535	1100	0.6368
0.8729	0.3856	1200	0.5956
1.0211	0.4177	1300	0.5921
0.8105	0.4499	1400	0.5674
0.8372	0.4820	1500	0.5341
0.9081	0.5141	1600	0.5218
0.8897	0.5463	1700	0.5129
0.8362	0.5784	1800	0.4881
0.8596	0.6105	1900	0.4853
0.7492	0.6427	2000	0.4792
0.8427	0.6748	2100	0.4608
0.8349	0.7069	2200	0.4568
0.7586	0.7391	2300	0.4544
0.7442	0.7712	2400	0.4516

Framework versions

• PEFT 0.12.1.dev0
• Transformers 4.44.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1