Hugging Face's logo

allen0909
/
ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V6

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V6 / README.md
allen0909's picture
allen0909
End of training
172aedb verified
preview code
|
raw
history blame contribute delete
6.51 kB
metadata
license: other
library_name: peft
tags:
  - trl
  - sft
  - generated_from_trainer
base_model: taide/TAIDE-LX-7B-Chat
model-index:
  - name: ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V6
    results: []

ROE_QA_TAIDE-LX-7B-Chat_Q100_80_20_V6

This model is a fine-tuned version of taide/TAIDE-LX-7B-Chat on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3442

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 1
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 4
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 5
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
4.8628 0.0321 100 3.4692
4.6437 0.0643 200 3.2308
3.8154 0.0964 300 2.8144
3.3006 0.1285 400 2.5033
2.953 0.1607 500 2.2690
2.2844 0.1928 600 1.9550
2.134 0.2249 700 1.7888
1.8182 0.2571 800 1.6310
1.6776 0.2892 900 1.5062
1.6317 0.3213 1000 1.4026
1.6311 0.3535 1100 1.2253
1.1003 0.3856 1200 1.0703
1.2014 0.4177 1300 1.0672
0.9489 0.4499 1400 0.9135
1.0068 0.4820 1500 0.8950
1.0342 0.5141 1600 0.8074
0.9713 0.5463 1700 0.7767
0.9598 0.5784 1800 0.6995
0.9357 0.6105 1900 0.6643
0.8298 0.6427 2000 0.6456
0.8803 0.6748 2100 0.6164
0.8899 0.7069 2200 0.5867
0.7877 0.7391 2300 0.5748
0.7792 0.7712 2400 0.5513
0.8387 0.8033 2500 0.5267
0.8625 0.8355 2600 0.5111
0.8514 0.8676 2700 0.4992
0.762 0.8997 2800 0.4881
0.779 0.9319 2900 0.4840
0.8938 0.9640 3000 0.4676
0.7007 0.9961 3100 0.4668
0.7006 1.0283 3200 0.4500
0.6622 1.0604 3300 0.4383
0.6724 1.0925 3400 0.4404
0.6551 1.1247 3500 0.4372
0.6395 1.1568 3600 0.4273
0.6317 1.1889 3700 0.4217
0.6683 1.2211 3800 0.4218
0.6987 1.2532 3900 0.4136
0.7309 1.2853 4000 0.4151
0.6192 1.3175 4100 0.4092
0.6013 1.3496 4200 0.4109
0.6138 1.3817 4300 0.4069
0.6224 1.4139 4400 0.4001
0.6414 1.4460 4500 0.3942
0.6234 1.4781 4600 0.3949
0.6365 1.5103 4700 0.3914
0.6602 1.5424 4800 0.3908
0.6646 1.5746 4900 0.3916
0.6295 1.6067 5000 0.3832
0.6379 1.6388 5100 0.3828
0.6544 1.6710 5200 0.3836
0.6224 1.7031 5300 0.3905
0.5892 1.7352 5400 0.3785
0.5985 1.7674 5500 0.3794
0.5776 1.7995 5600 0.3751
0.5799 1.8316 5700 0.3745
0.6247 1.8638 5800 0.3742
0.5724 1.8959 5900 0.3716
0.627 1.9280 6000 0.3750
0.566 1.9602 6100 0.3720
0.5754 1.9923 6200 0.3706
0.3561 2.0244 6300 0.3685
0.4468 2.0566 6400 0.3663
0.392 2.0887 6500 0.3669
0.4049 2.1208 6600 0.3659
0.3967 2.1530 6700 0.3649
0.418 2.1851 6800 0.3662
0.4973 2.2172 6900 0.3630
0.3685 2.2494 7000 0.3627
0.3967 2.2815 7100 0.3618
0.3921 2.3136 7200 0.3580
0.3905 2.3458 7300 0.3595
0.3888 2.3779 7400 0.3574
0.4403 2.4100 7500 0.3571
0.4343 2.4422 7600 0.3574
0.4088 2.4743 7700 0.3549
0.3924 2.5064 7800 0.3560
0.4383 2.5386 7900 0.3540
0.3786 2.5707 8000 0.3525
0.3779 2.6028 8100 0.3518
0.4555 2.6350 8200 0.3519
0.3813 2.6671 8300 0.3497
0.375 2.6992 8400 0.3505
0.3889 2.7314 8500 0.3493
0.3524 2.7635 8600 0.3469
0.431 2.7956 8700 0.3480
0.4185 2.8278 8800 0.3471
0.378 2.8599 8900 0.3487
0.3978 2.8920 9000 0.3457
0.421 2.9242 9100 0.3461
0.3691 2.9563 9200 0.3443
0.4019 2.9884 9300 0.3446
0.2392 3.0206 9400 0.3473
0.2213 3.0527 9500 0.3443
0.2189 3.0848 9600 0.3437
0.2363 3.1170 9700 0.3442

Framework versions

  • PEFT 0.12.1.dev0
  • Transformers 4.44.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1