Edit model card

pretrain_wo-cot_w-asd

This model is a fine-tuned version of EleutherAI/polyglot-ko-1.3b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 9.5465

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 3
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
11.1865 0.1290 1 11.1164
11.1109 0.2581 2 11.0354
11.0508 0.3871 3 10.9345
10.9774 0.5161 4 10.8247
10.8372 0.6452 5 10.7115
10.742 0.7742 6 10.5983
10.6067 0.9032 7 10.4797
10.5182 1.0323 8 10.3694
10.3704 1.1613 9 10.2614
10.2903 1.2903 10 10.1587
10.1627 1.4194 11 10.0626
10.1078 1.5484 12 9.9744
10.0265 1.6774 13 9.8932
9.9159 1.8065 14 9.8200
9.8432 1.9355 15 9.7532
9.7643 2.0645 16 9.6953
9.7308 2.1935 17 9.6462
9.6923 2.3226 18 9.6069
9.6348 2.4516 19 9.5776
9.5869 2.5806 20 9.5566
9.6044 2.7097 21 9.5465

Framework versions

  • PEFT 0.11.1
  • Transformers 4.41.1
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
8
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for slplab/pretrain_wo-cot_w-asd

Adapter
(25)
this model