See axolotl config

axolotl version: 0.4.1

strict: false

base_model: microsoft/Phi-3.5-mini-instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: true

chat_template: phi_3
datasets:
  - path: fozziethebeat/alpaca_messages_classifier_2k_test
    type: chat_template
    split: train
    chat_template: phi_3
    field_messages: messages
    message_field_role: role
    message_field_content: content
    roles:
      user:
        - user
      assistant:
        - assistant

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/lora-out

sequence_len: 2048
sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 8
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 5.0e-5

train_on_inputs: false
group_by_length: false
bfloat16: true
bf16: true
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
s2_attention:

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 4
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:

phi-3.5-alpaca-test-classifier

This model is a fine-tuned version of microsoft/Phi-3.5-mini-instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1174

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
11.7206	0.0187	1	11.9120
9.4452	0.2617	14	9.1059
2.2582	0.5234	28	1.8353
0.1463	0.7850	42	0.1658
0.1315	1.0467	56	0.1291
0.1207	1.3084	70	0.1218
0.1238	1.5701	84	0.1196
0.1103	1.8318	98	0.1174

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1