Hugging Face's logo

tommyp111
/
pythia-1.4b-deduped-alpaca-lora

PEFT
Safetensors
gpt_neox
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Community
pythia-1.4b-deduped-alpaca-lora / README.md
tommyp111's picture
tommyp111
Upload folder using huggingface_hub
944f94c verified
preview code
|
raw
history blame contribute delete
3.7 kB
metadata
base_model: EleutherAI/pythia-1.4b-deduped
library_name: peft
license: apache-2.0
tags:
  - generated_from_trainer
model-index:
  - name: outputs/lora-alpaca-pythia
    results: []

Built with Axolotl

See axolotl config

axolotl version: 0.4.1

base_model: EleutherAI/pythia-1.4b-deduped
load_in_8bit: true
datasets:
  - path: teknium/GPT4-LLM-Cleaned
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
adapter: lora
lora_model_dir:
sequence_len: 512
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - query_key_value
  - dense
  - dense_h_to_4h
  - dense_4h_to_h
lora_target_linear:
lora_fan_in_fan_out: true  # pythia/GPTNeoX lora specific
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
output_dir: ./outputs/lora-alpaca-pythia
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 4
learning_rate: 0.00001
train_on_inputs: false
group_by_length: false
bf16: auto
tf32: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
weight_decay: 0.1
evals_per_epoch: 4
logging_steps: 1
push_to_hub: tommyp111/pythia-1.4b-deduped-alpaca-lora
wandb_project: pythia-alpaca-lora
wandb_name: pythia-1.4b

Visualize in Weights & Biases

outputs/lora-alpaca-pythia

This model is a fine-tuned version of EleutherAI/pythia-1.4b-deduped on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.2444

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 4

Training results

Training Loss Epoch Step Validation Loss
1.5363 0.0001 1 2.6674
1.2474 0.25 3190 1.4167
1.3556 0.5 6380 1.3586
1.2912 0.75 9570 1.3302
1.2149 1.0 12760 1.3089
1.6017 1.25 15950 1.2917
1.1827 1.5 19140 1.2827
0.9565 1.75 22330 1.2739
1.2363 2.0 25520 1.2674
1.3477 2.25 28710 1.2596
1.6589 2.5 31900 1.2571
1.1538 2.75 35090 1.2530
1.5866 3.0 38280 1.2473
1.0768 3.25 41470 1.2464
1.4019 3.5 44660 1.2452
1.1724 3.75 47850 1.2434
1.3227 4.0 51040 1.2444

Framework versions

  • PEFT 0.11.1
  • Transformers 4.43.1
  • Pytorch 2.3.1+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1