Add interleaved and video/3d prompt

README.md

@@ -83,6 +83,26 @@ output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
 print(processor.decode(output[0][2:], skip_special_tokens=True))
 ```
 
+When prompting with videos/3D data/multi-view data, prompt like following: 
+
+```python
+# if you downsampled n frames from the input
+
+image_tokens = "<image>" * n
+prompt = f"<|im_start|>user {image_tokens}\nWhat are these?|im_end|><|im_start|>assistant"
+```
+
+When prompting with interleaved images and videos, prompt like following: 
+
+```python
+# two interleaved images
+prompt = "<|im_start|>user <image><image>\nWhat are these?|im_end|><|im_start|>assistant"
+
+# two interleaved videos, if you downsampled n frames in total from both videos
+image_tokens = "<image>" * n
+prompt = f"<|im_start|>user {image_tokens}\nWhat are these?|im_end|><|im_start|>assistant"
+```
+
 ### Model optimization
 
 #### 4-bit quantization through `bitsandbytes` library