wizardLM-LlaMA-LoRA-7B

A LoRA trained on the WizardLM dataset, with a LlaMA 7B as the basemodel.

Instruction example

It was trained with the alpaca-short template, without any inputs, so prompt as follows:

### Instruction:
Write a poem about the transformers Python library. 
Mention the word "large language models" in that poem.
### Response:
I'm not sure if this is what you meant, but here goes!
The Transformers are large language models
that help us make sense of text.
They take our sentences and turn them into vectors,
which can be used to find similarities between texts.
We use these for things like search engines or spam filters;
they also have uses in machine learning too.

Trained with the following params

base_model: /root/alpaca-lora/llama-7b-hf
data_path: victor123/evol_instruct_70k
output_dir: /loras/wizardLM-lama-lora
batch_size: 64
micro_batch_size: 8
num_epochs: 3
learning_rate: 2e-05
cutoff_len: 2048
val_set_size: 2000
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
train_on_inputs: True
add_eos_token: False
group_by_length: True
wandb_project: 
wandb_run_name: 
wandb_watch: 
wandb_log_model: 
resume_from_checkpoint: False
prompt template: alpaca_short

Training Details

Trained with https://github.com/tloen/alpaca-lora. Note: ince the victor123/evol_instruct_70k dataset only contains instruction and output, comment out the line data_point["input"], around line 151 in alpaca-lora\finetune.py
Trained on RunPod community cloud with 1x A100 80GB vram (Note: less GPU was needed)
Took 14:47:39 to train 3 epochs
Cost around $37 to train

Evaluation

No evaluation has been done on this model. If someone wants to share I would happily pull.
Empirically it looks promising for complex instruction following.