Update README.md

README.md

@@ -1,3 +1,119 @@
 ---
-license: cc-by-4.0
+license: openrail
+datasets:
+- NbAiLab/norwegian-alpaca
+language:
+- 'no'
+- nb
+pipeline_tag: text-generation
 ---
+
+# NB-Alpaca-LoRA 7B
+
+This is an Norwegian adapter generated by fine-tuning LLaMA-7B on a [Norwegian Alpaca](https://huggingface.co/datasets/NbAiLab/norwegian-alpaca) dataset.
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import LLaMATokenizer, LLaMAForCausalLM
+
+base_model = "decapoda-research/llama-7b-hf"
+tokenizer = LLaMATokenizer.from_pretrained(base_model)
+model = LLaMAForCausalLM.from_pretrained(
+    base_model,
+    load_in_8bit=True,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(model, "NbAiLab/nb-alpaca-lora-7b")
+```
+
+For generation, the promtp still needs the English template:
+
+```python
+from transformers import pipeline
+
+pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
+instruction = "Skriv en e-post der du ønsker velkommen til en ny medarbeider ved navn Svein"
+pipe.generate(f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+
+### Instruction:
+{instruction}
+
+### Response:
+""")
+# Kjære Svein,
+#
+# Velkommen til vårt team! Vi er så glade for å ha deg med oss. Vi ser frem til å hjelpe deg med å nå dine mål og oppnå dine drømmer.
+#
+# Vi er alltid tilgjengelige hvis du har noen spørsmål eller ønsker å diskutere noen av våre prosjekter.
+#
+# Vi ser frem til å jobbe sammen med deg!
+#
+# Med vennlig
+```
+
+
+## Data
+
+The dataset is a translation to Norwegian Bokmål of [alpaca_data_cleaned.json](https://github.com/tloen/alpaca-lora/blob/main/alpaca_data_cleaned.json) (a clean version of the [Alpaca dataset made at Stanford](https://huggingface.co/datasets/tatsu-lab/alpaca)) using OpenAI's `gpt-3.5-turbo` model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of `(intruction, input, output)` and costed around $60.0.
+
+**This dataset cannot be used to create models that compete in any way with OpenAI.**
+
+## Finetuning
+
+To fine-tune the LLaMA model we used the code available on [Alpaca Lora](https://github.com/tloen/alpaca-lora), which provides code to finetune the LLaMA model using PEFT from Hugging Face. We run finetuning for 3 epochs using sequence length of 512 with no gradient accumulation on a single A6000 with 48GB VRAM for 12 hours on top of LLaMA-7B.
+
+## Example outputs
+
+Stanford Alpaca:
+```
+Instruction: Kom med en kreativ unnskyldning for å si at jeg ikke trenger å gå på festen.
+Response: I'm sorry, but I don't need to go to the party.
+```
+
+NB-Alpaca:
+```
+Instruction: Kom med en kreativ unnskyldning for å si at jeg ikke trenger å gå på festen.
+Response: Jeg beklager at jeg ikke kan gå på festen, men jeg tror at det er bedre at jeg ikke gjør det.
+```
+
+
+Stanford Alpaca:
+```
+Instruction: Hvordan kan jeg starte en karriere som dataforsker? Skriv i listeform.
+Response:
+Starting a career as a data scientist requires a combination of education, experience, and skills. 
+
+1. Obtain a Bachelor's or Master's degree in computer science, statistics, mathematics, or a related field. 
+2. Gain practical experience through internships or part-time jobs. 
+3. Learn programming languages such as Python, R, and SQL. 
+4. Develop strong analytical and problem-solving skills. 
+5. Build a portfolio of projects to showcase your work. 
+6. Network with other data scientists to learn
+```
+
+NB-Alpaca:
+```
+Instruction: Hvordan kan jeg starte en karriere som dataforsker? Skriv i listeform.
+Response:
+1. Søk etter en universitetsgrad i datavitenskap, matematikk, informatikk eller en relatert fag. 
+2. Søk etter en jobb som dataforsker eller dataanalytiker. 
+3. Utvikle dine ferdigheter i datavitenskap, matematikk, informatikk eller en relatert fag. 
+4. Utvikle dine ferdigheter innen dataanalyse.
+```
+
+You can test it using the eval notebook [here](https://colab.research.google.com/github/22-hours/cabrita/blob/main/notebooks/cabrita-lora.ipynb).
+
+## References
+
+- [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/)
+- [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
+- [Norwegian Alpaca](https://huggingface.co/datasets/NbAiLab/norwegian-alpaca)
+- [Alpaca LoRA](https://github.com/tloen/alpaca-lora)
+- [ChatGPT](https://openai.com/blog/chatgpt)
+- [Hugging Face](https://huggingface.co/)
+ 
+## Hardware Requirements
+
+For training we have used an A6000 48GB VRAM Nvidia GPU. For eval, you can use a T4.