gemma-2b-dolly-qa

This model is a fine-tuned version of google/gemma-2b on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 2.1557

Model description

Fine-tuning learning progress.

Training and evaluation data

Fine tuned on databricks-dolly-15k with LoRa

Training procedure

Everything took place on the Intel Developer Cloud. Was fine tuned on Intel(R) Data Center GPU Max 1100.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• training_steps: 593

Training results

Training Loss	Epoch	Step	Validation Loss
2.8555	0.82	100	2.5404
2.4487	1.64	200	2.3221
2.3054	2.46	300	2.2302
2.2362	3.28	400	2.1790
2.197	4.1	500	2.1557

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.0.1a0+cxx11.abi
• Datasets 2.18.0
• Tokenizers 0.15.2