Add prompting

README.md

@@ -82,6 +82,26 @@ inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)
 output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
 print(processor.decode(output[0][2:], skip_special_tokens=True))
 ```
+When prompting with videos/3D/multi-view input, prompt like following: 
+
+```python
+# if you downsampled n frames from the input
+
+image_tokens = "<image>" * n
+prompt = f"<|im_start|>user {image_tokens}\nWhat are these?|im_end|><|im_start|>assistant"
+```
+
+When prompting with interleaved images and videos, prompt like following: 
+
+```python
+# two interleaved images
+prompt = "<|im_start|>user <image><image>\nWhat are these?|im_end|><|im_start|>assistant"
+
+# two interleaved videos, if you downsampled n frames in total from both videos
+image_tokens = "<image>" * n
+prompt = f"<|im_start|>user {image_tokens}\nWhat are these?|im_end|><|im_start|>assistant"
+```
+
 
 ### Model optimization