init commit

README.md

@@ -1,3 +1,61 @@
----
-license: gpl-3.0
----
+# wizardLM-LlaMA-LoRA-7B
+
+A LoRA trained on the WizardLM dataset, with a LlaMA 7B as the basemodel.
+
+## Instruction example
+
+It was trained with the alpaca-short template, without any inputs, so prompt as follows:
+
+```
+### Instruction:
+Write a poem about the transformers Python library. 
+Mention the word "large language models" in that poem.
+### Response:
+I'm not sure if this is what you meant, but here goes!
+The Transformers are large language models
+that help us make sense of text.
+They take our sentences and turn them into vectors,
+which can be used to find similarities between texts.
+We use these for things like search engines or spam filters;
+they also have uses in machine learning too.
+```
+
+## Trained with the following params
+
+```
+base_model: /root/alpaca-lora/llama-7b-hf
+data_path: victor123/evol_instruct_70k
+output_dir: /loras/wizardLM-lama-lora
+batch_size: 64
+micro_batch_size: 8
+num_epochs: 3
+learning_rate: 2e-05
+cutoff_len: 2048
+val_set_size: 2000
+lora_r: 16
+lora_alpha: 16
+lora_dropout: 0.05
+lora_target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
+train_on_inputs: True
+add_eos_token: False
+group_by_length: True
+wandb_project: 
+wandb_run_name: 
+wandb_watch: 
+wandb_log_model: 
+resume_from_checkpoint: False
+prompt template: alpaca_short
+```
+
+## Training Details
+
+- Trained with https://github.com/tloen/alpaca-lora. Note: ince the `victor123/evol_instruct_70k` dataset only contains instruction and output, comment out the line `data_point["input"],` around line 151 in `alpaca-lora\finetune.py`
+- Trained on [RunPod](https://runpod.io?ref=qgrfwczf
+) community cloud with 1x A100 80GB vram (Note: less GPU was needed)
+- Took 14:47:39 to train 3 epochs
+- Cost around $37 to train
+
+## Evaluation
+
+- No evaluation has been done on this model. If someone wants to share I would happily pull.
+- Empirically it looks promising for complex instruction following.

adapter_config.json

@@ -0,0 +1,19 @@
+{
+  "base_model_name_or_path": "/root/alpaca-lora/llama-7b-hf",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "target_modules": [
+    "q_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5e1621f48d9ad8feb1d6d31050275f0aafd080c5c07153301fe2f48411f4406
+size 443