update readme

README.md

@@ -4,10 +4,9 @@ language:
 - en
 ---
 
-# RedPajama-Chat-INCITE-6.9B
+# RedPajama-Instruct-INCITE-6.9B
 
-RedPajama-Chat-INCITE-6.9B-v1, is a large transformer-based language model developed by Together Computer and trained on the RedPajama-Data-1T dataset.
-It is further fine-tuned on GPT-JT's datasets enhance zero/few-shot in-context learning.
+RedPajama-Instruct-INCITE-6.9B-v1, is a large transformer-based language model developed by Together Computer and trained on the RedPajama-Data-1T dataset.
 
 ## Model Details
 - **Developed by**: Together Computer.
@@ -18,52 +17,115 @@ It is further fine-tuned on GPT-JT's datasets enhance zero/few-shot in-context l
 
 # Quick Start
 
+Please note that the model requires `transformers` version >= 4.25.1.
+
 ## GPU Inference
 
-This requires a GPU with 16GB memory.
+This requires a GPU with 8GB memory.
+
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1", torch_dtype=torch.float16)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1", torch_dtype=torch.float16)
 model = model.to('cuda:0')
 # infer
-inputs = tokenizer("Hello", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
 ## GPU Inference in Int8
 
-This requires a GPU with 12GB memory.
+This requires a GPU with 6GB memory.
+
+To run inference with int8, please ensure you have installed accelerate and bitandbytes. You can install them with the following command:
+
+```bash
+pip install accelerate
+pip install bitsandbytes
+```
+
+Then you can run inference with int8 as follows:
 
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1", device_map="auto", load_in_8bit=True)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1", device_map='auto', torch_dtype=torch.float16, load_in_8bit=True)
+
 # infer
-inputs = tokenizer("Hello", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
 ## CPU Inference
 
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-6.9B-v1", torch_dtype=torch.bfloat16)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-6.9B-v1", torch_dtype=torch.bfloat16)
 # infer
-inputs = tokenizer("<human>: Hello!\n<bot>:", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
+Please note that since `LayerNormKernelImpl` is not implemented in fp16 for CPU, we use `bfloat16` for CPU inference.
+
 
 # Uses
 
@@ -85,7 +147,7 @@ It is the responsibility of the end user to ensure that the model is used in a r
 
 #### Out-of-Scope Use
 
-RedPajama-Chat-INCITE-6.9B is a language model and may not perform well for other use cases outside of its intended scope. 
+RedPajama-Instruct-INCITE-6.9B is a language model and may not perform well for other use cases outside of its intended scope. 
 For example, it may not be suitable for use in safety-critical applications or for making decisions that have a significant impact on individuals or society. 
 It is important to consider the limitations of the model and to only use it for its intended purpose.
 
@@ -108,7 +170,7 @@ Using the model to generate content that is cruel to individuals is a misuse of
 
 ## Limitations
 
-RedPajama-Chat-INCITE-6.9B, like other language models, has limitations that should be taken into consideration. 
+RedPajama-Instruct-INCITE-6.9B, like other language models, has limitations that should be taken into consideration. 
 For example, the model may not always provide accurate or relevant answers, particularly for questions that are complex, ambiguous, or outside of its training data. 
 We therefore welcome contributions from individuals and organizations, and encourage collaboration towards creating a more robust and inclusive chatbot.
 
@@ -123,7 +185,7 @@ Please refer to [togethercomputer/RedPajama-Data-1T](https://huggingface.co/data
 - **Hardware:** 8 A100
 - **Optimizer:** Adam
 - **Gradient Accumulations**: 1
-- **Num of Tokens:** 0.8B Tokens
+- **Num of Tokens:** 131M tokens
 - **Learning rate:** 1e-5
 
 ## Community