distilgpt2-fables-demo

Training: The model has been trained using the script provided in the following repository https://github.com/MorenoLaQuatra/transformers-tasks-templates

This model is a fine-tuned version of distilgpt2 on demelin/understanding_fables dataset. It achieves the following results on the evaluation set:

• Loss: 3.2165

Model description

The model is a demo for the fine-tuning of decoder-only models using transformers library.

Intended uses & limitations

It can be used mainly for prototyping and educational purposes.

Training and evaluation data

The demelin/understanding_fables dataset has been split into train/test/validation using an 80/10/10 random split (random_seed = 42). Google Colab has been used for model fine-tuning.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	38	42.4563
No log	2.0	76	5.2808
28.753	3.0	114	3.7712
28.753	4.0	152	3.4577
28.753	5.0	190	3.3120
3.5846	6.0	228	3.2773
3.5846	7.0	266	3.2710
3.0017	8.0	304	3.2764
3.0017	9.0	342	3.2795
3.0017	10.0	380	3.3300

Framework versions

• Transformers 4.22.1
• Pytorch 1.12.1+cu113
• Datasets 2.4.0
• Tokenizers 0.12.1