Update README.md

README.md

@@ -6,7 +6,7 @@ datasets:
 - instruction-pretrain/ft-instruction-synthesizer-collection
 ---
 # Instruction Pre-Training: Language Models are Supervised Multitask Learners
-This repo contains the **context-based instruction synthesizer** used in our paper **Instruction Pre-Training: Language Models are Supervised Multitask Learners**.
+This repo contains the **context-based instruction synthesizer** in our paper **Instruction Pre-Training: Language Models are Supervised Multitask Learners**.
 
 We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.
 
@@ -28,7 +28,7 @@ We explore supervised multitask pre-training by proposing ***Instruction Pre-Tra
   - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B)
 
 
-## Synthesize Instruction-Response Pairs based on Any Raw text
+## Synthesize Instruction-Response Pairs to Augment Any Raw Corpora
 We conduct multitask fine-tuning on a language model to develop an instruction synthesizer capable of generating instruction-response pairs from any raw text. The fine-tuning data are available at [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
 
 
@@ -36,7 +36,7 @@ We conduct multitask fine-tuning on a language model to develop an instruction s
     <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/0889QyG59QM3rPeZlcTzZ.png" width="700">
 </p>
 
-For example, to prompt the synthesizer to generate instruction-response pairs based on a given raw text:
+### Basic Usage: Synthesize instruction-response pairs based on a given raw text
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
@@ -75,13 +75,13 @@ def get_instruction_response_pairs(context):
     '''Prompt the synthesizer to generate instruction-response pairs based on the given context'''
     prompt = f'<s> <CON> {context} </CON>\n\n'
     inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
-    outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]
+    outputs = model.generate(input_ids=inputs, max_new_tokens=400, do_sample=False)[0]
 
     pred_start = int(inputs.shape[-1])
     pred = tokenizer.decode(outputs[pred_start:], skip_special_tokens=True)
     return parse_pred(pred)
 
-# Get the list of generated instruction-response paris
+# Get the generated instruction-response paris
 instruction_response_pairs = get_instruction_response_pairs(context)
 
 # Print out the results
@@ -90,8 +90,106 @@ for index, pair in enumerate(instruction_response_pairs):
     print(f'## Instruction {index + 1}:\n{pair["Q"]}\n## Response {index + 1}:\n{pair["A"]}\n')
 ```
 
-### To-Do
-- [ ] Add example usages for synthesizing few-shot examples
+### Advanced Usage: Synthesize Few-shot Examples
+A one-shot example consists of a piece of raw text followed by its instruction-response pairs. You can conduct multi-round inferece to synthesize a few-shot example: the instruction-response pairs of different raw texts share the same pattern.
+
+To accelerate synthesis, we use the [vLLM framework](https://github.com/vllm-project/vllm?tab=readme-ov-file):
+<details>
+<summary> Click to expand </summary>
+
+1. Set up dependencies:
+Install vLLM with pip or from [source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
+
+```bash
+pip install vllm
+```
+
+2. Synthesize:
+```python
+from vllm import LLM, SamplingParams
+
+# Put your list of raw texts here, 
+# a list of M raw texts can be coverted into an M-shot example:
+text_list = [
+    "Genetically and medically susceptible workers.\nThe likelihood of an individual becoming ill from a hazardous material or condition is strongly influenced by both their genetic makeup and their underlying state of health. Although the past decade has seen great advances in understanding human variation in health and genetic polymorphisms and in the diagnosis and treatment of disease, much less progress has been made in effectively using this information to protect worker health. Scientific evidence for increased susceptibility often is weak and rarely satisfies legal thresholds for sufficient risk to warrant exclusion from a particular job. When public safety is a major concern, many legally mandated exclusions are not well justified. Medical opinions about fitness to work should be based upon a systematic and credible analysis of the condition, its relationship to ability and risk for a particular job, and knowledge of possible accommodations. Conclusions should reflect the limitations of scientific knowledge and guidance from antidiscrimination legislation.", 
+    "Exclusive Breastfeeding for Twin Babies and Its Influencing Factors: A Study in East Java, Indonesia.\nThis study aimed to identify the factors that influence the success of exclusive breastfeeding in twins. This cross-sectional study was conducted on 184 mothers who had twins aged 6-23 months in Malang Raya, East Java, Indonesia and used the consecutive sampling technique. The data was collected through distributing questionnaires containing questions related to knowledge about exclusive breastfeeding, breastfeeding self-efficacy, and the support of family and certified health workers. Multinomial regression statistical test results show that the most influential factor for the success of exclusive breastfeeding with twins was breastfeeding self-efficacy (OR 0.111; 95% CI 0.033-0.387). A high level of breastfeeding self-efficacy can increase a mother's confidence to be able to provide exclusive breastfeeding for twins. This study suggests that nurses can provide breastfeeding counselling to improve breastfeeding self-efficacy."]
+
+# Create a sampling params object.
+sampling_params = SamplingParams(temperature=0, max_tokens=400)
+
+# Load the model and tokenizer
+llm = LLM(model="instruction-pretrain/instruction-synthesizer", max_model_len=4096)
+
+# Templates (please do NOT change them)
+context_template = ' <CON> {context} </CON>'
+QA_template = '<QUE> {question} <ANS> {answer} </END>'
+delimiter = '\n\n'
+bos_token = '<s>'
+eos_token = '</s>'
+
+def cook_context(raw_context):
+    """Format the context."""
+    return bos_token + context_template.replace('{context}', raw_context) + delimiter
+
+def cook_instruction_response_pairs(QA_list):
+    """Format downstream instruction(Q)-response(A) pairs."""
+    ins_res_list = []
+    for qa_entry in QA_list:
+        qa = QA_template.replace('{question}', qa_entry['Q']).replace('{answer}', qa_entry['A'])
+        ins_res_list.append(qa)
+    return delimiter.join(ins_res_list) + eos_token
+
+def parse_pred(pred):
+    """Extract the list of instruction-response pairs from the prediction"""
+    QA_str_list = pred.split('</END>')
+    if not pred.endswith('</END>'):
+        QA_str_list = QA_str_list[:-1]
+
+    QA_list = []
+    raw_questions = []
+    for QA_str in QA_str_list:
+        try:
+            assert len(QA_str.split('<ANS>')) == 2, f'invalid QA string: {QA_str}'
+            Q_str, A_str = QA_str.split('<ANS>')
+            Q_str, A_str = Q_str.strip(), A_str.strip()
+            assert Q_str.startswith('<QUE>'), f'invalid question string: {Q_str} in QA_str: {QA_str}'
+            assert len(A_str) > 0, f'invalid answer string in QA_str: {QA_str}'
+            Q_str = Q_str.replace('<QUE>', '').strip()
+            assert Q_str.lower() not in raw_questions, f'duplicate question: {Q_str}'
+            QA_list.append({'Q': Q_str, 'A': A_str})
+            raw_questions.append(Q_str.lower())
+        except:
+            pass
+
+    return QA_list
+
+def get_instruction_response_pairs(context):
+    '''Prompt the synthesizer to generate instruction-response pairs based on the given context'''
+    outputs = llm.generate(context, sampling_params, use_tqdm=False)
+    pred = outputs[0].outputs[0].text
+    return parse_pred(pred)
+
+# Process each text and generate instruction-response pairs in multi-round inference:
+previous_examples = []
+for cur_text in text_list:
+    # Prepend raw texts and instruction-response pairs of previous examples to the current text
+    context = ''
+    for previous_example in previous_examples:
+        context += cook_context(previous_example['text']) + cook_instruction_response_pairs(previous_example['instruction_response_pairs'])
+    context += cook_context(cur_text)
+
+    # Get the generated instruction-response paris
+    instruction_response_pairs = get_instruction_response_pairs(context)
+    previous_examples.append({'text': cur_text, 'instruction_response_pairs': instruction_response_pairs})
+
+# Concatenate the raw texts and instruction-response pairs of M rounds to consititute an M-shot example
+for example in previous_examples:
+    print(f'# Raw Text:\n{example["text"]}\n')
+    for index, pair in enumerate(example['instruction_response_pairs']):
+        print(f'## Instruction {index + 1}:\n{pair["Q"]}\n## Response {index + 1}:\n{pair["A"]}\n')
+```
+</details>
+
 
 ## Citation
 If you find our work helpful, please cite us: