Update README.md

README.md

@@ -6,7 +6,7 @@ datasets:
 - instruction-pretrain/ft-instruction-synthesizer-collection
 ---
 # Instruction Pre-Training: Language Models are Supervised Multitask Learners
-This repo contains the **context-based instruction synthesizer** in our paper **Instruction Pre-Training: Language Models are Supervised Multitask Learners**.
+This repo contains the **context-based instruction synthesizer** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
 
 We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.