Update README.md

README.md

@@ -37,7 +37,8 @@ The base model [braindao/flan-t5-cnn](https://huggingface.co/braindao/flan-t5-cn
 ## Model description
 
 * This model further finetuned [braindao/flan-t5-cnn](https://huggingface.co/braindao/flan-t5-cnn) on the more conversational samsum dataset.  
-* Huggingface [PEFT Library](https://github.com/huggingface/peft) LoRA (r = 16) was used to further reduced the model size.  Only 1.7M parameters were trained (0.71% of original flan-t5-base 250M parameters).
+* Huggingface [PEFT Library](https://github.com/huggingface/peft) LoRA (r = 16) and bitsandbytes int-8 was used to speed up training and reduce the model size.
+* Only 1.7M parameters were trained (0.71% of original flan-t5-base 250M parameters).
 * The model checkpoint is just 7MB.
 
 ## Intended uses & limitations
@@ -61,6 +62,36 @@ The following hyperparameters were used during training:
 - rougeL: 37.300937%
 - rougeLsum: 37.271341%
 
+### How to use
+
+```python
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+# Load peft config for pre-trained checkpoint etc.
+peft_model_id = "sooolee/flan-t5-base-cnn-samsum-lora"
+config = PeftConfig.from_pretrained(peft_model_id)
+
+# load base LLM model and tokenizer
+model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path, load_in_8bit=True, device_map='auto')
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+
+# Load the Lora model
+model = PeftModel.from_pretrained(model, peft_model_id, device_map='auto')
+
+# Tokenize the text inputs
+texts = "<e.g. Part of YouTube Transcript>"
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+with torch.no_grad():    
+    output = self.model.generate(input_ids=inputs["input_ids"].to(device), max_new_tokens=60, do_sample=True, top_p=0.9)
+    summary = self.tokenizer.batch_decode(output.detach().cpu().numpy(), skip_special_tokens=True)
+
+summary
+```
+
 ### Framework versions
 
 - Transformers 4.27.2