# wizardLM-LlaMA-LoRA-7B

A LoRA trained on the WizardLM dataset, with a LlaMA 7B as the basemodel.

## Instruction example

It was trained with the alpaca-short template, without any inputs, so prompt as follows:

```
### Instruction:
Write a poem about the transformers Python library. 
Mention the word "large language models" in that poem.
### Response:
I'm not sure if this is what you meant, but here goes!
The Transformers are large language models
that help us make sense of text.
They take our sentences and turn them into vectors,
which can be used to find similarities between texts.
We use these for things like search engines or spam filters;
they also have uses in machine learning too.
```

## Trained with the following params

```
base_model: /root/alpaca-lora/llama-7b-hf
data_path: victor123/evol_instruct_70k
output_dir: /loras/wizardLM-lama-lora
batch_size: 64
micro_batch_size: 8
num_epochs: 3
learning_rate: 2e-05
cutoff_len: 2048
val_set_size: 2000
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
train_on_inputs: True
add_eos_token: False
group_by_length: True
wandb_project: 
wandb_run_name: 
wandb_watch: 
wandb_log_model: 
resume_from_checkpoint: False
prompt template: alpaca_short
```

## Training Details

- Trained with https://github.com/tloen/alpaca-lora. Note: ince the `victor123/evol_instruct_70k` dataset only contains instruction and output, comment out the line `data_point["input"],` around line 151 in `alpaca-lora\finetune.py`
- Trained on [RunPod](https://runpod.io?ref=qgrfwczf
) community cloud with 1x A100 80GB vram (Note: less GPU was needed)
- Took 14:47:39 to train 3 epochs
- Cost around $37 to train

## Evaluation

- No evaluation has been done on this model. If someone wants to share I would happily pull.
- Empirically it looks promising for complex instruction following.