pinzhenchen commited on
Commit
2357370
1 Parent(s): 57b41f8

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +40 -0
README.md ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+ language:
4
+ - fi
5
+ tags:
6
+ - generation
7
+ - question answering
8
+ - instruction tuning
9
+ license: cc-by-nc-4.0
10
+ ---
11
+
12
+ ### Model Description
13
+
14
+ This HF repository contains base LLMs instruction tuned (SFT) with LoRA and then used to study whether monolingual or multilingual instruction tuning is more favourable.
15
+ * [GitHub](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main)
16
+ * [Paper](https://arxiv.org/abs/2309.08958)
17
+
18
+ #### Instruction tuning details
19
+ * Base model: [EleutherAI/pythia-1.4b-deduped](https://huggingface.co/EleutherAI/pythia-1.4b-deduped)
20
+ * Instruction tuning language: Finnish
21
+ * Training method: LoRA.
22
+ * LoRA details: rank=8, alpha=16, target modules={key, query, value}.
23
+ * Best checkpoint: best cross-entropy on a validation set, trained for 5 epochs.
24
+ * Dataset: machine-translated from [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned). You can download our data [HERE](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main/training-data).
25
+
26
+ #### Usage
27
+ The model checkpoint should be loaded with the base model together using `transformers` and `peft` libraries.
28
+
29
+ Please refer to our Github repository [HERE](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main/loraft) for inference and training instructions.
30
+
31
+ #### Citation
32
+ ```
33
+ @inproceedings{chen-etal-2024-monolingual,
34
+ title="Monolingual or multilingual instruction tuning: Which makes a better {Alpaca}",
35
+ author="Pinzhen Chen and Shaoxiong Ji and Nikolay Bogoychev and Andrey Kutuzov and Barry Haddow and Kenneth Heafield",
36
+ year="2024",
37
+ booktitle = "Findings of the Association for Computational Linguistics: EACL 2024",
38
+ }
39
+ ```
40
+