Model Card for Llama-3.1-8B-KAM

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the None dataset.

Model description

More information needed

Quick start

from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="MaRyAm1295/Llama-3.1-8B-KAM", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training procedure

This model was trained with SFT.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 3407
• gradient_accumulation_steps: 16
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 20
• training_steps: 500
• mixed_precision_training: Native AMP

Training results

Step    Training Loss

• 50     2.158200
• 100     1.845900
• 150     1.832200
• 200     1.805300
• 250     1.783800
• 300     1.767500
• 350     1.744800
• 400     1.745600
• 450     1.749500
• 500     1.756100

Framework versions

• TRL: 0.12.0
• Transformers: 4.46.2
• Pytorch: 2.4.0
• Datasets: 3.0.1
• Tokenizers: 0.20.0