czczup commited on
Commit
c9be6a0
1 Parent(s): a257b0d

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ examples/red-panda.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,643 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: image-text-to-text
4
+ ---
5
+
6
+ # InternVL2-40B
7
+
8
+ [\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL) [\[🆕 Blog\]](https://internvl.github.io/blog/) [\[📜 InternVL 1.0 Paper\]](https://arxiv.org/abs/2312.14238) [\[📜 InternVL 1.5 Report\]](https://arxiv.org/abs/2404.16821)
9
+
10
+ [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/675877376)
11
+
12
+ ## Introduction
13
+
14
+ We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 2 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-40B model.
15
+
16
+ Compared to the state-of-the-art open-source multimodal large language models, InternVL 2.0 surpasses most open-source models. It demonstrates competitive performance on par with proprietary commercial models across various capabilities, including document and chart comprehension, infographics QA, scene text understanding and OCR tasks, scientific and mathematical problem solving, as well as cultural understanding and integrated multimodal capabilities.
17
+
18
+ InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
19
+
20
+ ## Model Details
21
+
22
+ InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-40B consists of [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5), an MLP projector, and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B).
23
+
24
+ ## Performance
25
+
26
+ ### Image Benchmarks
27
+
28
+ | Benchmark | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
29
+ | :--------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
30
+ | Model Size | - | - | 25.5B | 40B |
31
+ | | | | | |
32
+ | DocVQA<sub>test</sub> | 87.2 | 86.5 | 92.9 | |
33
+ | ChartQA<sub>test</sub> | 78.1 | 81.3 | 84.9 | 86.2 |
34
+ | InfoVQA<sub>test</sub> | - | 72.7 | 75.9 | |
35
+ | TextVQA<sub>val</sub> | - | 73.5 | 82.3 | 83.0 |
36
+ | OCRBench | 678 | 754 | 825 | 837 |
37
+ | MME<sub>sum</sub> | 2070.2 | 2110.6 | 2260.7 | 2315.0 |
38
+ | RealWorldQA | 68.0 | 67.5 | 68.3 | 71.8 |
39
+ | AI2D<sub>test</sub> | 89.4 | 80.3 | 84.5 | 87.1 |
40
+ | MMMU<sub>val</sub> | 63.1 | 58.5 | 48.3 | 53.9 |
41
+ | MMBench-EN<sub>test</sub> | 81.0 | 73.9 | 83.4 | 86.8 |
42
+ | MMBench-CN<sub>test</sub> | 80.2 | 73.8 | 82.0 | 86.5 |
43
+ | CCBench<sub>dev</sub> | 57.3 | 28.4 | 73.5 | 80.6 |
44
+ | MMVet<sub>GPT-4-0613</sub> | - | - | 64.2 | 68.5 |
45
+ | MMVet<sub>GPT-4-Turbo</sub> | 67.5 | 64.0 | 62.1 | 65.5 |
46
+ | SEED-Image | - | - | 76.8 | 78.2 |
47
+ | HallBench<sub>avg</sub> | 43.9 | 45.6 | 50.7 | 56.9 |
48
+ | MathVista<sub>testmini</sub> | 58.1 | 57.7 | 59.4 | 63.7 |
49
+
50
+ - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. MMMU, OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
51
+
52
+ - Please note that evaluating the same model using different testing toolkits like InternVL and VLMEvalKit can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.
53
+
54
+ - It is important to mention that the MMVet scores we report are evaluated using GPT-4-0613 as the judge model. Different versions of GPT-4 can lead to significant variations in the scores for this dataset. For instance, using GPT-4-Turbo would result in significantly lower scores.
55
+
56
+ ### Video Benchmarks
57
+
58
+ | Benchmark | GPT-4V | VILA-1.5 | LLaVA-NeXT-Video | InternVL2-26B | InternVL2-40B |
59
+ | :------------------: | :----: | :------: | :--------------: | :-----------: | :-----------: |
60
+ | Model Size | - | 34B | 34B | 25.5B | 40B |
61
+ | | | | | | |
62
+ | MVBench | - | - | - | 67.5 | 72.5 |
63
+ | Video-MME<br>wo subs | 59.9 | 59.0 | 52.0 | TBD | TBD |
64
+ | Video-MME<br>w/ subs | 63.3 | 59.4 | 54.9 | TBD | TBD |
65
+
66
+ - We evaluate our models on MVBench by extracting 16 frames from each video, and each frame was resized to a 448x448 image.
67
+
68
+ Limitations: Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.
69
+
70
+ ## Quick Start
71
+
72
+ We provide an example code to run InternVL2-40B using `transformers`.
73
+
74
+ > Please use transformers==4.37.2 to ensure the model works normally.
75
+
76
+ ```python
77
+ import numpy as np
78
+ import torch
79
+ import torchvision.transforms as T
80
+ from decord import VideoReader, cpu
81
+ from PIL import Image
82
+ from torchvision.transforms.functional import InterpolationMode
83
+ from transformers import AutoModel, AutoTokenizer
84
+
85
+ IMAGENET_MEAN = (0.485, 0.456, 0.406)
86
+ IMAGENET_STD = (0.229, 0.224, 0.225)
87
+
88
+
89
+ def build_transform(input_size):
90
+ MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
91
+ transform = T.Compose([
92
+ T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
93
+ T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
94
+ T.ToTensor(),
95
+ T.Normalize(mean=MEAN, std=STD)
96
+ ])
97
+ return transform
98
+
99
+
100
+ def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
101
+ best_ratio_diff = float('inf')
102
+ best_ratio = (1, 1)
103
+ area = width * height
104
+ for ratio in target_ratios:
105
+ target_aspect_ratio = ratio[0] / ratio[1]
106
+ ratio_diff = abs(aspect_ratio - target_aspect_ratio)
107
+ if ratio_diff < best_ratio_diff:
108
+ best_ratio_diff = ratio_diff
109
+ best_ratio = ratio
110
+ elif ratio_diff == best_ratio_diff:
111
+ if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
112
+ best_ratio = ratio
113
+ return best_ratio
114
+
115
+
116
+ def dynamic_preprocess(image, min_num=1, max_num=6, image_size=448, use_thumbnail=False):
117
+ orig_width, orig_height = image.size
118
+ aspect_ratio = orig_width / orig_height
119
+
120
+ # calculate the existing image aspect ratio
121
+ target_ratios = set(
122
+ (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
123
+ i * j <= max_num and i * j >= min_num)
124
+ target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
125
+
126
+ # find the closest aspect ratio to the target
127
+ target_aspect_ratio = find_closest_aspect_ratio(
128
+ aspect_ratio, target_ratios, orig_width, orig_height, image_size)
129
+
130
+ # calculate the target width and height
131
+ target_width = image_size * target_aspect_ratio[0]
132
+ target_height = image_size * target_aspect_ratio[1]
133
+ blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
134
+
135
+ # resize the image
136
+ resized_img = image.resize((target_width, target_height))
137
+ processed_images = []
138
+ for i in range(blocks):
139
+ box = (
140
+ (i % (target_width // image_size)) * image_size,
141
+ (i // (target_width // image_size)) * image_size,
142
+ ((i % (target_width // image_size)) + 1) * image_size,
143
+ ((i // (target_width // image_size)) + 1) * image_size
144
+ )
145
+ # split the image
146
+ split_img = resized_img.crop(box)
147
+ processed_images.append(split_img)
148
+ assert len(processed_images) == blocks
149
+ if use_thumbnail and len(processed_images) != 1:
150
+ thumbnail_img = image.resize((image_size, image_size))
151
+ processed_images.append(thumbnail_img)
152
+ return processed_images
153
+
154
+
155
+ def load_image(image_file, input_size=448, max_num=6):
156
+ image = Image.open(image_file).convert('RGB')
157
+ transform = build_transform(input_size=input_size)
158
+ images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
159
+ pixel_values = [transform(image) for image in images]
160
+ pixel_values = torch.stack(pixel_values)
161
+ return pixel_values
162
+
163
+
164
+ path = 'OpenGVLab/InternVL2-40B'
165
+ # If you have an 80G A100 GPU, you can put the entire model on a single GPU.
166
+ model = AutoModel.from_pretrained(
167
+ path,
168
+ torch_dtype=torch.bfloat16,
169
+ low_cpu_mem_usage=True,
170
+ trust_remote_code=True).eval().cuda()
171
+ # Otherwise, you need to set device_map='auto' to use multiple GPUs for inference.
172
+ # import os
173
+ # os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
174
+ # model = AutoModel.from_pretrained(
175
+ # path,
176
+ # torch_dtype=torch.bfloat16,
177
+ # low_cpu_mem_usage=True,
178
+ # trust_remote_code=True,
179
+ # device_map='auto').eval()
180
+
181
+ tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
182
+ # set the max number of tiles in `max_num`
183
+ pixel_values = load_image('./examples/image1.jpg', max_num=6).to(torch.bfloat16).cuda()
184
+
185
+ generation_config = dict(
186
+ num_beams=1,
187
+ max_new_tokens=1024,
188
+ do_sample=False,
189
+ )
190
+
191
+ # pure-text conversation (纯文本对话)
192
+ question = 'Hello, who are you?'
193
+ response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
194
+ print(f'User: {question}')
195
+ print(f'Assistant: {response}')
196
+
197
+ question = 'Can you tell me a story?'
198
+ response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
199
+ print(f'User: {question}')
200
+ print(f'Assistant: {response}')
201
+
202
+ # single-image single-round conversation (单图单轮对话)
203
+ question = '<image>\nPlease describe the image shortly.'
204
+ response = model.chat(tokenizer, pixel_values, question, generation_config)
205
+ print(f'User: {question}')
206
+ print(f'Assistant: {response}')
207
+
208
+ # single-image multi-round conversation (单图多轮对话)
209
+ question = '<image>\nPlease describe the image in detail.'
210
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
211
+ print(f'User: {question}')
212
+ print(f'Assistant: {response}')
213
+
214
+ question = 'Please write a poem according to the image.'
215
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
216
+ print(f'User: {question}')
217
+ print(f'Assistant: {response}')
218
+
219
+ # multi-image multi-round conversation, combined images (多图多轮对话,拼接图像)
220
+ pixel_values1 = load_image('./examples/image1.jpg', max_num=6).to(torch.bfloat16).cuda()
221
+ pixel_values2 = load_image('./examples/image2.jpg', max_num=6).to(torch.bfloat16).cuda()
222
+ pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
223
+
224
+ question = '<image>\nDescribe the two images in detail.'
225
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
226
+ history=None, return_history=True)
227
+
228
+ question = 'What are the similarities and differences between these two images.'
229
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
230
+ history=history, return_history=True)
231
+ print(f'User: {question}')
232
+ print(f'Assistant: {response}')
233
+
234
+ # multi-image multi-round conversation, separate images (多图多轮对话,独立图像)
235
+ pixel_values1 = load_image('./examples/image1.jpg', max_num=6).to(torch.bfloat16).cuda()
236
+ pixel_values2 = load_image('./examples/image2.jpg', max_num=6).to(torch.bfloat16).cuda()
237
+ pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
238
+ num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
239
+
240
+ question = 'Image-1: <image>\nImage-2: <image>\nDescribe the two images in detail.'
241
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
242
+ num_patches_list=num_patches_list,
243
+ history=None, return_history=True)
244
+ print(f'User: {question}')
245
+ print(f'Assistant: {response}')
246
+
247
+ question = 'What are the similarities and differences between these two images.'
248
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
249
+ num_patches_list=num_patches_list,
250
+ history=history, return_history=True)
251
+ print(f'User: {question}')
252
+ print(f'Assistant: {response}')
253
+
254
+ # batch inference, single image per sample (单图批处理)
255
+ pixel_values1 = load_image('./examples/image1.jpg', max_num=6).to(torch.bfloat16).cuda()
256
+ pixel_values2 = load_image('./examples/image2.jpg', max_num=6).to(torch.bfloat16).cuda()
257
+ num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
258
+ pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
259
+
260
+ questions = ['<image>\nDescribe the image in detail.'] * len(num_patches_list)
261
+ responses = model.batch_chat(tokenizer, pixel_values,
262
+ num_patches_list=num_patches_list,
263
+ questions=questions,
264
+ generation_config=generation_config)
265
+ for question, response in zip(questions, responses):
266
+ print(f'User: {question}')
267
+ print(f'Assistant: {response}')
268
+
269
+ # video multi-round conversation (视频多轮对话)
270
+ def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):
271
+ if bound:
272
+ start, end = bound[0], bound[1]
273
+ else:
274
+ start, end = -100000, 100000
275
+ start_idx = max(first_idx, round(start * fps))
276
+ end_idx = min(round(end * fps), max_frame)
277
+ seg_size = float(end_idx - start_idx) / num_segments
278
+ frame_indices = np.array([
279
+ int(start_idx + (seg_size / 2) + np.round(seg_size * idx))
280
+ for idx in range(num_segments)
281
+ ])
282
+ return frame_indices
283
+
284
+ def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):
285
+ vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
286
+ max_frame = len(vr) - 1
287
+ fps = float(vr.get_avg_fps())
288
+
289
+ pixel_values_list, num_patches_list = [], []
290
+ transform = build_transform(input_size=input_size)
291
+ frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)
292
+ for frame_index in frame_indices:
293
+ img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')
294
+ img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)
295
+ pixel_values = [transform(tile) for tile in img]
296
+ pixel_values = torch.stack(pixel_values)
297
+ num_patches_list.append(pixel_values.shape[0])
298
+ pixel_values_list.append(pixel_values)
299
+ pixel_values = torch.cat(pixel_values_list)
300
+ return pixel_values, num_patches_list
301
+
302
+
303
+ video_path = './examples/red-panda.mp4'
304
+ # pixel_values, num_patches_list = load_video(video_path, num_segments=32, max_num=1)
305
+ pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)
306
+ pixel_values = pixel_values.to(torch.bfloat16).cuda()
307
+ video_prefix = ''.join([f'Frame{i+1}: <image>\n' for i in range(len(num_patches_list))])
308
+ question = video_prefix + 'What is the red panda doing?'
309
+ # Frame1: <image>\nFrame2: <image>\n...\nFrame31: <image>\n{question}
310
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
311
+ num_patches_list=num_patches_list,
312
+ history=None, return_history=True)
313
+ print(f'User: {question}')
314
+ print(f'Assistant: {response}')
315
+
316
+ question = 'Describe this video in detail. Don\'t repeat.'
317
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config,
318
+ num_patches_list=num_patches_list,
319
+ history=history, return_history=True)
320
+ print(f'User: {question}')
321
+ print(f'Assistant: {response}')
322
+ ```
323
+
324
+ ## Deployment
325
+
326
+ ### LMDeploy
327
+
328
+ LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams.
329
+
330
+ ```sh
331
+ pip install lmdeploy
332
+ ```
333
+
334
+ LMDeploy abstracts the complex inference process of multi-modal Vision-Language Models (VLM) into an easy-to-use pipeline, similar to the Large Language Model (LLM) inference pipeline.
335
+
336
+ #### A 'Hello, world' example
337
+
338
+ ```python
339
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
340
+ from lmdeploy.vl import load_image
341
+
342
+ model = 'OpenGVLab/InternVL2-40B'
343
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
344
+ image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
345
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
346
+ chat_template_config.meta_instruction = system_prompt
347
+ pipe = pipeline(model, chat_template_config=chat_template_config,
348
+ backend_config=TurbomindEngineConfig(session_len=8192))
349
+ response = pipe(('describe this image', image))
350
+ print(response.text)
351
+ ```
352
+
353
+ If `ImportError` occurs while executing this case, please install the required dependency packages as prompted.
354
+
355
+ #### Multi-images inference
356
+
357
+ When dealing with multiple images, you can put them all in one list. Keep in mind that multiple images will lead to a higher number of input tokens, and as a result, the size of the context window typically needs to be increased.
358
+
359
+ ```python
360
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
361
+ from lmdeploy.vl import load_image
362
+ from lmdeploy.vl.constants import IMAGE_TOKEN
363
+
364
+ model = 'OpenGVLab/InternVL2-40B'
365
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
366
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
367
+ chat_template_config.meta_instruction = system_prompt
368
+ pipe = pipeline(model, chat_template_config=chat_template_config,
369
+ backend_config=TurbomindEngineConfig(session_len=8192))
370
+
371
+ image_urls=[
372
+ 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg',
373
+ 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg'
374
+ ]
375
+
376
+ images = [load_image(img_url) for img_url in image_urls]
377
+ # Numbering images improves multi-image conversations
378
+ response = pipe((f'Image-1: {IMAGE_TOKEN}\nImage-2: {IMAGE_TOKEN}\ndescribe these two images', images))
379
+ print(response.text)
380
+ ```
381
+
382
+ #### Batch prompts inference
383
+
384
+ Conducting inference with batch prompts is quite straightforward; just place them within a list structure:
385
+
386
+ ```python
387
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
388
+ from lmdeploy.vl import load_image
389
+
390
+ model = 'OpenGVLab/InternVL2-40B'
391
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
392
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
393
+ chat_template_config.meta_instruction = system_prompt
394
+ pipe = pipeline(model, chat_template_config=chat_template_config,
395
+ backend_config=TurbomindEngineConfig(session_len=8192))
396
+
397
+ image_urls=[
398
+ "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg",
399
+ "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg"
400
+ ]
401
+ prompts = [('describe this image', load_image(img_url)) for img_url in image_urls]
402
+ response = pipe(prompts)
403
+ print(response)
404
+ ```
405
+
406
+ #### Multi-turn conversation
407
+
408
+ There are two ways to do the multi-turn conversations with the pipeline. One is to construct messages according to the format of OpenAI and use above introduced method, the other is to use the `pipeline.chat` interface.
409
+
410
+ ```python
411
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig, GenerationConfig
412
+ from lmdeploy.vl import load_image
413
+
414
+ model = 'OpenGVLab/InternVL2-40B'
415
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
416
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
417
+ chat_template_config.meta_instruction = system_prompt
418
+ pipe = pipeline(model, chat_template_config=chat_template_config,
419
+ backend_config=TurbomindEngineConfig(session_len=8192))
420
+
421
+ image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg')
422
+ gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.8)
423
+ sess = pipe.chat(('describe this image', image), gen_config=gen_config)
424
+ print(sess.response.text)
425
+ sess = pipe.chat('What is the woman doing?', session=sess, gen_config=gen_config)
426
+ print(sess.response.text)
427
+ ```
428
+
429
+ ## License
430
+
431
+ This project is released under the MIT license, while InternLM is licensed under the Apache-2.0 license.
432
+
433
+ ## Citation
434
+
435
+ If you find this project useful in your research, please consider citing:
436
+
437
+ ```BibTeX
438
+ @article{chen2023internvl,
439
+ title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
440
+ author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
441
+ journal={arXiv preprint arXiv:2312.14238},
442
+ year={2023}
443
+ }
444
+ @article{chen2024far,
445
+ title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
446
+ author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
447
+ journal={arXiv preprint arXiv:2404.16821},
448
+ year={2024}
449
+ }
450
+ ```
451
+
452
+ ## 简介
453
+
454
+ 我们很高兴宣布 InternVL 2.0 的发布,这是 InternVL 系列多模态大语言模型的最新版本。InternVL 2.0 提供了多种**指令微调**的模型,参数从 20 亿到 1080 亿不等。此仓库包含经过指令微调的 InternVL2-40B 模型。
455
+
456
+ 与最先进的开源多模态大语言模型相比,InternVL 2.0 超越了大多数开源模型。它在各种能力上表现出与闭源商业模型相媲美的竞争力,包括文档和图表理解、信息图表问答、场景文本理解和 OCR 任务、科学和数学问题解决,以及文化理解和综合多模态能力。
457
+
458
+ InternVL 2.0 使用 8k 上下文窗口进行训练,训练数据包含长文本、多图和视频数据,与 InternVL 1.5 相比,其处理这些类型输入的能力显著提高。更多详细信息,请参阅我们的博客和 GitHub。
459
+
460
+ ## 模型细节
461
+
462
+ InternVL 2.0 是一个多模态大语言模型系列,包含各种规模的模型。对于每个规模的模型,我们都会发布针对多模态任务优化的指令微调模型。InternVL2-40B 包含 [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5)、一个 MLP 投影器和 [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)。
463
+
464
+ ## 性能测试
465
+
466
+ ### 图像相关评测
467
+
468
+ | 评测数据集 | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
469
+ | :--------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
470
+ | 模型大小 | - | - | 25.5B | 40B |
471
+ | | | | | |
472
+ | DocVQA<sub>test</sub> | 87.2 | 86.5 | 92.9 | |
473
+ | ChartQA<sub>test</sub> | 78.1 | 81.3 | 84.9 | 86.2 |
474
+ | InfoVQA<sub>test</sub> | - | 72.7 | 75.9 | |
475
+ | TextVQA<sub>val</sub> | - | 73.5 | 82.3 | 83.0 |
476
+ | OCRBench | 678 | 754 | 825 | 837 |
477
+ | MME<sub>sum</sub> | 2070.2 | 2110.6 | 2260.7 | 2315.0 |
478
+ | RealWorldQA | 68.0 | 67.5 | 68.3 | 71.8 |
479
+ | AI2D<sub>test</sub> | 89.4 | 80.3 | 84.5 | 87.1 |
480
+ | MMMU<sub>val</sub> | 63.1 | 58.5 | 48.3 | 53.9 |
481
+ | MMBench-EN<sub>test</sub> | 81.0 | 73.9 | 83.4 | 86.8 |
482
+ | MMBench-CN<sub>test</sub> | 80.2 | 73.8 | 82.0 | 86.5 |
483
+ | CCBench<sub>dev</sub> | 57.3 | 28.4 | 73.5 | 80.6 |
484
+ | MMVet<sub>GPT-4-0613</sub> | - | - | 64.2 | 68.5 |
485
+ | MMVet<sub>GPT-4-Turbo</sub> | 67.5 | 64.0 | 62.1 | 65.5 |
486
+ | SEED-Image | - | - | 76.8 | 78.2 |
487
+ | HallBench<sub>avg</sub> | 43.9 | 45.6 | 50.7 | 56.9 |
488
+ | MathVista<sub>testmini</sub> | 58.1 | 57.7 | 59.4 | 63.7 |
489
+
490
+ - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说,DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
491
+
492
+ - 请注意,使用不同的测试工具包(如 InternVL 和 VLMEvalKit)评估同一模型可能会导致细微差异,这是正常的。代码版本的更新、环境和硬件的变化也可能导致结果的微小差异。
493
+
494
+ - 需要提到的是,我们报告的 MMVet 分数是使用 GPT-4-0613 作为评判模型评估的。不同版本的 GPT-4 会导致该数据集分数的显著变化。例如,使用 GPT-4-Turbo 会导致分数显著降低。
495
+
496
+ ### 视频相关评测
497
+
498
+ | 评测数据集 | GPT-4V | VILA-1.5 | LLaVA-NeXT-Video | InternVL2-26B | InternVL2-40B |
499
+ | :------------------: | :----: | :------: | :--------------: | :-----------: | :-----------: |
500
+ | 模型大小 | - | 34B | 34B | 25.5B | 40B |
501
+ | | | | | | |
502
+ | MVBench | - | - | - | 67.5 | 72.5 |
503
+ | Video-MME<br>wo subs | 59.9 | 59.0 | 52.0 | TBD | TBD |
504
+ | Video-MME<br>w/ subs | 63.3 | 59.4 | 54.9 | TBD | TBD |
505
+
506
+ - 我们通过从每个视频中提取16帧来评估我们的模型在MVBench上的性能,每个视频帧被调整为448x448的图像。
507
+
508
+ 限制:尽管在训练过程中我们非常注重模型的安全性,尽力促使模型输出符合伦理和法律要求的文本,但受限于模型大小以及概率生成范式,模型可能会产生各种不符合预期的输出,例如回复内容包含偏见、歧视等有害内容,请勿传播这些内容。由于传播不良信息导致的任何后果,本项目不承担责任。
509
+
510
+ ## 快速启动
511
+
512
+ 我们提供了一个示例代码,用于使用 `transformers` 运行 InternVL2-40B。
513
+
514
+ > 请使用 transformers==4.37.2 以确保模型正常运行。
515
+
516
+ 示例代码请[点击这里](#quick-start)。
517
+
518
+ ## 部署
519
+
520
+ ### LMDeploy
521
+
522
+ LMDeploy 是由 MMRazor 和 MMDeploy 团队开发的用于压缩、部署和服务大语言模型(LLM)的工具包。
523
+
524
+ ```sh
525
+ pip install lmdeploy
526
+ ```
527
+
528
+ LMDeploy 将多模态视觉-语言模型(VLM)的复杂推理过程抽象为一个易于使用的管道,类似于大语言模型(LLM)的推理管道。
529
+
530
+ #### 一个“你好,世界”示例
531
+
532
+ ```python
533
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
534
+ from lmdeploy.vl import load_image
535
+
536
+ model = 'OpenGVLab/InternVL2-40B'
537
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和���业发展。'
538
+ image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
539
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
540
+ chat_template_config.meta_instruction = system_prompt
541
+ pipe = pipeline(model, chat_template_config=chat_template_config,
542
+ backend_config=TurbomindEngineConfig(session_len=8192))
543
+ response = pipe(('describe this image', image))
544
+ print(response.text)
545
+ ```
546
+
547
+ 如果在执行此示例时出现 `ImportError`,请按照提示安装所需的依赖包。
548
+
549
+ #### 多图像推理
550
+
551
+ 在处理多张图像时,可以将它们全部放入一个列表中。请注意,多张图像会导致输入 token 数量增加,因此通常需要增加上下文窗口的大小。
552
+
553
+ ```python
554
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
555
+ from lmdeploy.vl import load_image
556
+ from lmdeploy.vl.constants import IMAGE_TOKEN
557
+
558
+ model = 'OpenGVLab/InternVL2-40B'
559
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
560
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
561
+ chat_template_config.meta_instruction = system_prompt
562
+ pipe = pipeline(model, chat_template_config=chat_template_config,
563
+ backend_config=TurbomindEngineConfig(session_len=8192))
564
+
565
+ image_urls=[
566
+ 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg',
567
+ 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg'
568
+ ]
569
+
570
+ images = [load_image(img_url) for img_url in image_urls]
571
+ response = pipe((f'Image-1: {IMAGE_TOKEN}\nImage-2: {IMAGE_TOKEN}\ndescribe these two images', images))
572
+ print(response.text)
573
+ ```
574
+
575
+ #### 批量Prompt推理
576
+
577
+ 使用批量Prompt进行推理非常简单;只需将它们放在一个列表结构中:
578
+
579
+ ```python
580
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
581
+ from lmdeploy.vl import load_image
582
+
583
+ model = 'OpenGVLab/InternVL2-40B'
584
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
585
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
586
+ chat_template_config.meta_instruction = system_prompt
587
+ pipe = pipeline(model, chat_template_config=chat_template_config,
588
+ backend_config=TurbomindEngineConfig(session_len=8192))
589
+
590
+ image_urls=[
591
+ "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg",
592
+ "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg"
593
+ ]
594
+ prompts = [('describe this image', load_image(img_url)) for img_url in image_urls]
595
+ response = pipe(prompts)
596
+ print(response)
597
+ ```
598
+
599
+ #### 多轮对话
600
+
601
+ 使用管道进行多轮对话有两种方法。一种是根据 OpenAI 的格式构建消息并使用上述方法,另一种是使用 `pipeline.chat` 接口。
602
+
603
+ ```python
604
+ from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig, GenerationConfig
605
+ from lmdeploy.vl import load_image
606
+
607
+ model = 'OpenGVLab/InternVL2-40B'
608
+ system_prompt = '我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。'
609
+ chat_template_config = ChatTemplateConfig('internvl-internlm2')
610
+ chat_template_config.meta_instruction = system_prompt
611
+ pipe = pipeline(model, chat_template_config=chat_template_config,
612
+ backend_config=TurbomindEngineConfig(session_len=8192))
613
+
614
+ image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg')
615
+ gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.8)
616
+ sess = pipe.chat(('describe this image', image), gen_config=gen_config)
617
+ print(sess.response.text)
618
+ sess = pipe.chat('What is the woman doing?', session=sess, gen_config=gen_config)
619
+ print(sess.response.text)
620
+ ```
621
+
622
+ ## 开源许可证
623
+
624
+ 该项目采用 MIT 许可证发布,而 InternLM 则采用 Apache-2.0 许可证。
625
+
626
+ ## 引用
627
+
628
+ 如果您发现此项目对您的研究有用,可以考虑引用我们的论文:
629
+
630
+ ```BibTeX
631
+ @article{chen2023internvl,
632
+ title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
633
+ author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
634
+ journal={arXiv preprint arXiv:2312.14238},
635
+ year={2023}
636
+ }
637
+ @article{chen2024far,
638
+ title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
639
+ author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
640
+ journal={arXiv preprint arXiv:2404.16821},
641
+ year={2024}
642
+ }
643
+ ```
added_tokens.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</box>": 64006,
3
+ "</quad>": 64002,
4
+ "</ref>": 64004,
5
+ "<IMG_CONTEXT>": 64000,
6
+ "<box>": 64005,
7
+ "<quad>": 64001,
8
+ "<ref>": 64003
9
+ }
config.json ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_commit_hash": null,
3
+ "architectures": [
4
+ "InternVLChatModel"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
8
+ "AutoModel": "modeling_internvl_chat.InternVLChatModel",
9
+ "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
10
+ },
11
+ "downsample_ratio": 0.5,
12
+ "dynamic_image_size": true,
13
+ "force_image_size": 448,
14
+ "llm_config": {
15
+ "_name_or_path": "NousResearch/Nous-Hermes-2-Yi-34B",
16
+ "add_cross_attention": false,
17
+ "architectures": [
18
+ "LlamaForCausalLM"
19
+ ],
20
+ "attention_bias": false,
21
+ "attention_dropout": 0.0,
22
+ "bad_words_ids": null,
23
+ "begin_suppress_tokens": null,
24
+ "bos_token_id": 1,
25
+ "chunk_size_feed_forward": 0,
26
+ "cross_attention_hidden_size": null,
27
+ "decoder_start_token_id": null,
28
+ "diversity_penalty": 0.0,
29
+ "do_sample": false,
30
+ "early_stopping": false,
31
+ "encoder_no_repeat_ngram_size": 0,
32
+ "eos_token_id": 7,
33
+ "exponential_decay_length_penalty": null,
34
+ "finetuning_task": null,
35
+ "forced_bos_token_id": null,
36
+ "forced_eos_token_id": null,
37
+ "hidden_act": "silu",
38
+ "hidden_size": 7168,
39
+ "id2label": {
40
+ "0": "LABEL_0",
41
+ "1": "LABEL_1"
42
+ },
43
+ "initializer_range": 0.02,
44
+ "intermediate_size": 20480,
45
+ "is_decoder": false,
46
+ "is_encoder_decoder": false,
47
+ "label2id": {
48
+ "LABEL_0": 0,
49
+ "LABEL_1": 1
50
+ },
51
+ "length_penalty": 1.0,
52
+ "max_length": 20,
53
+ "max_position_embeddings": 8192,
54
+ "min_length": 0,
55
+ "model_type": "llama",
56
+ "no_repeat_ngram_size": 0,
57
+ "num_attention_heads": 56,
58
+ "num_beam_groups": 1,
59
+ "num_beams": 1,
60
+ "num_hidden_layers": 60,
61
+ "num_key_value_heads": 8,
62
+ "num_return_sequences": 1,
63
+ "output_attentions": false,
64
+ "output_hidden_states": false,
65
+ "output_scores": false,
66
+ "pad_token_id": 0,
67
+ "prefix": null,
68
+ "pretraining_tp": 1,
69
+ "problem_type": null,
70
+ "pruned_heads": {},
71
+ "remove_invalid_values": false,
72
+ "repetition_penalty": 1.0,
73
+ "return_dict": true,
74
+ "return_dict_in_generate": false,
75
+ "rms_norm_eps": 1e-05,
76
+ "rope_scaling": {
77
+ "factor": 3.0,
78
+ "type": "dynamic"
79
+ },
80
+ "rope_theta": 5000000.0,
81
+ "sep_token_id": null,
82
+ "suppress_tokens": null,
83
+ "task_specific_params": null,
84
+ "temperature": 1.0,
85
+ "tf_legacy_loss": false,
86
+ "tie_encoder_decoder": false,
87
+ "tie_word_embeddings": false,
88
+ "tokenizer_class": null,
89
+ "top_k": 50,
90
+ "top_p": null,
91
+ "torch_dtype": "bfloat16",
92
+ "torchscript": false,
93
+ "transformers_version": "4.37.2",
94
+ "typical_p": 1.0,
95
+ "use_bfloat16": true,
96
+ "use_cache": true,
97
+ "vocab_size": 64007
98
+ },
99
+ "max_dynamic_patch": 12,
100
+ "min_dynamic_patch": 1,
101
+ "model_type": "internvl_chat",
102
+ "ps_version": "v2",
103
+ "select_layer": -1,
104
+ "template": "Hermes-2",
105
+ "torch_dtype": "bfloat16",
106
+ "use_backbone_lora": 0,
107
+ "use_llm_lora": 0,
108
+ "use_thumbnail": true,
109
+ "vision_config": {
110
+ "architectures": [
111
+ "InternVisionModel"
112
+ ],
113
+ "attention_dropout": 0.0,
114
+ "drop_path_rate": 0.0,
115
+ "dropout": 0.0,
116
+ "hidden_act": "gelu",
117
+ "hidden_size": 3200,
118
+ "image_size": 448,
119
+ "initializer_factor": 0.1,
120
+ "initializer_range": 1e-10,
121
+ "intermediate_size": 12800,
122
+ "layer_norm_eps": 1e-06,
123
+ "model_type": "intern_vit_6b",
124
+ "norm_type": "rms_norm",
125
+ "num_attention_heads": 25,
126
+ "num_channels": 3,
127
+ "num_hidden_layers": 45,
128
+ "output_attentions": false,
129
+ "output_hidden_states": false,
130
+ "patch_size": 14,
131
+ "qk_normalization": true,
132
+ "qkv_bias": false,
133
+ "return_dict": true,
134
+ "torch_dtype": "bfloat16",
135
+ "transformers_version": "4.37.2",
136
+ "use_bfloat16": true,
137
+ "use_flash_attn": true
138
+ }
139
+ }
configuration_intern_vit.py ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # InternVL
3
+ # Copyright (c) 2023 OpenGVLab
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # --------------------------------------------------------
6
+ import os
7
+ from typing import Union
8
+
9
+ from transformers.configuration_utils import PretrainedConfig
10
+ from transformers.utils import logging
11
+
12
+ logger = logging.get_logger(__name__)
13
+
14
+
15
+ class InternVisionConfig(PretrainedConfig):
16
+ r"""
17
+ This is the configuration class to store the configuration of a [`InternVisionModel`]. It is used to
18
+ instantiate a vision encoder according to the specified arguments, defining the model architecture.
19
+
20
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
21
+ documentation from [`PretrainedConfig`] for more information.
22
+
23
+ Args:
24
+ num_channels (`int`, *optional*, defaults to 3):
25
+ Number of color channels in the input images (e.g., 3 for RGB).
26
+ patch_size (`int`, *optional*, defaults to 14):
27
+ The size (resolution) of each patch.
28
+ image_size (`int`, *optional*, defaults to 224):
29
+ The size (resolution) of each image.
30
+ qkv_bias (`bool`, *optional*, defaults to `False`):
31
+ Whether to add a bias to the queries and values in the self-attention layers.
32
+ hidden_size (`int`, *optional*, defaults to 3200):
33
+ Dimensionality of the encoder layers and the pooler layer.
34
+ num_attention_heads (`int`, *optional*, defaults to 25):
35
+ Number of attention heads for each attention layer in the Transformer encoder.
36
+ intermediate_size (`int`, *optional*, defaults to 12800):
37
+ Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
38
+ qk_normalization (`bool`, *optional*, defaults to `True`):
39
+ Whether to normalize the queries and keys in the self-attention layers.
40
+ num_hidden_layers (`int`, *optional*, defaults to 48):
41
+ Number of hidden layers in the Transformer encoder.
42
+ use_flash_attn (`bool`, *optional*, defaults to `True`):
43
+ Whether to use flash attention mechanism.
44
+ hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
45
+ The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
46
+ `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
47
+ layer_norm_eps (`float`, *optional*, defaults to 1e-6):
48
+ The epsilon used by the layer normalization layers.
49
+ dropout (`float`, *optional*, defaults to 0.0):
50
+ The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
51
+ drop_path_rate (`float`, *optional*, defaults to 0.0):
52
+ Dropout rate for stochastic depth.
53
+ attention_dropout (`float`, *optional*, defaults to 0.0):
54
+ The dropout ratio for the attention probabilities.
55
+ initializer_range (`float`, *optional*, defaults to 0.02):
56
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
57
+ initializer_factor (`float`, *optional*, defaults to 0.1):
58
+ A factor for layer scale.
59
+ """
60
+
61
+ model_type = 'intern_vit_6b'
62
+
63
+ def __init__(
64
+ self,
65
+ num_channels=3,
66
+ patch_size=14,
67
+ image_size=224,
68
+ qkv_bias=False,
69
+ hidden_size=3200,
70
+ num_attention_heads=25,
71
+ intermediate_size=12800,
72
+ qk_normalization=True,
73
+ num_hidden_layers=48,
74
+ use_flash_attn=True,
75
+ hidden_act='gelu',
76
+ norm_type='rms_norm',
77
+ layer_norm_eps=1e-6,
78
+ dropout=0.0,
79
+ drop_path_rate=0.0,
80
+ attention_dropout=0.0,
81
+ initializer_range=0.02,
82
+ initializer_factor=0.1,
83
+ **kwargs,
84
+ ):
85
+ super().__init__(**kwargs)
86
+
87
+ self.hidden_size = hidden_size
88
+ self.intermediate_size = intermediate_size
89
+ self.dropout = dropout
90
+ self.drop_path_rate = drop_path_rate
91
+ self.num_hidden_layers = num_hidden_layers
92
+ self.num_attention_heads = num_attention_heads
93
+ self.num_channels = num_channels
94
+ self.patch_size = patch_size
95
+ self.image_size = image_size
96
+ self.initializer_range = initializer_range
97
+ self.initializer_factor = initializer_factor
98
+ self.attention_dropout = attention_dropout
99
+ self.layer_norm_eps = layer_norm_eps
100
+ self.hidden_act = hidden_act
101
+ self.norm_type = norm_type
102
+ self.qkv_bias = qkv_bias
103
+ self.qk_normalization = qk_normalization
104
+ self.use_flash_attn = use_flash_attn
105
+
106
+ @classmethod
107
+ def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> 'PretrainedConfig':
108
+ config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
109
+
110
+ if 'vision_config' in config_dict:
111
+ config_dict = config_dict['vision_config']
112
+
113
+ if 'model_type' in config_dict and hasattr(cls, 'model_type') and config_dict['model_type'] != cls.model_type:
114
+ logger.warning(
115
+ f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
116
+ f'{cls.model_type}. This is not supported for all configurations of models and can yield errors.'
117
+ )
118
+
119
+ return cls.from_dict(config_dict, **kwargs)
configuration_internvl_chat.py ADDED
@@ -0,0 +1,93 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # InternVL
3
+ # Copyright (c) 2023 OpenGVLab
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # --------------------------------------------------------
6
+
7
+ import copy
8
+
9
+ from transformers import AutoConfig, LlamaConfig
10
+ from transformers.configuration_utils import PretrainedConfig
11
+ from transformers.utils import logging
12
+
13
+ from .configuration_intern_vit import InternVisionConfig
14
+
15
+ logger = logging.get_logger(__name__)
16
+
17
+
18
+ class InternVLChatConfig(PretrainedConfig):
19
+ model_type = 'internvl_chat'
20
+ is_composition = True
21
+
22
+ def __init__(
23
+ self,
24
+ vision_config=None,
25
+ llm_config=None,
26
+ use_backbone_lora=0,
27
+ use_llm_lora=0,
28
+ select_layer=-1,
29
+ force_image_size=None,
30
+ downsample_ratio=0.5,
31
+ template=None,
32
+ dynamic_image_size=False,
33
+ use_thumbnail=False,
34
+ ps_version='v1',
35
+ min_dynamic_patch=1,
36
+ max_dynamic_patch=6,
37
+ **kwargs):
38
+ super().__init__(**kwargs)
39
+
40
+ if vision_config is None:
41
+ vision_config = {}
42
+ logger.info('vision_config is None. Initializing the InternVisionConfig with default values.')
43
+
44
+ if llm_config is None:
45
+ llm_config = {}
46
+ logger.info('llm_config is None. Initializing the LlamaConfig config with default values (`LlamaConfig`).')
47
+
48
+ self.vision_config = InternVisionConfig(**vision_config)
49
+ if llm_config['architectures'][0] == 'LlamaForCausalLM':
50
+ self.llm_config = LlamaConfig(**llm_config)
51
+ else:
52
+ raise ValueError('Unsupported architecture: {}'.format(llm_config['architectures'][0]))
53
+ self.use_backbone_lora = use_backbone_lora
54
+ self.use_llm_lora = use_llm_lora
55
+ self.select_layer = select_layer
56
+ self.force_image_size = force_image_size
57
+ self.downsample_ratio = downsample_ratio
58
+ self.template = template
59
+ self.dynamic_image_size = dynamic_image_size
60
+ self.use_thumbnail = use_thumbnail
61
+ self.ps_version = ps_version # pixel shuffle version
62
+ self.min_dynamic_patch = min_dynamic_patch
63
+ self.max_dynamic_patch = max_dynamic_patch
64
+
65
+ logger.info(f'vision_select_layer: {self.select_layer}')
66
+ logger.info(f'ps_version: {self.ps_version}')
67
+ logger.info(f'min_dynamic_patch: {self.min_dynamic_patch}')
68
+ logger.info(f'max_dynamic_patch: {self.max_dynamic_patch}')
69
+
70
+ def to_dict(self):
71
+ """
72
+ Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`].
73
+
74
+ Returns:
75
+ `Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
76
+ """
77
+ output = copy.deepcopy(self.__dict__)
78
+ output['vision_config'] = self.vision_config.to_dict()
79
+ output['llm_config'] = self.llm_config.to_dict()
80
+ output['model_type'] = self.__class__.model_type
81
+ output['use_backbone_lora'] = self.use_backbone_lora
82
+ output['use_llm_lora'] = self.use_llm_lora
83
+ output['select_layer'] = self.select_layer
84
+ output['force_image_size'] = self.force_image_size
85
+ output['downsample_ratio'] = self.downsample_ratio
86
+ output['template'] = self.template
87
+ output['dynamic_image_size'] = self.dynamic_image_size
88
+ output['use_thumbnail'] = self.use_thumbnail
89
+ output['ps_version'] = self.ps_version
90
+ output['min_dynamic_patch'] = self.min_dynamic_patch
91
+ output['max_dynamic_patch'] = self.max_dynamic_patch
92
+
93
+ return output
conversation.py ADDED
@@ -0,0 +1,390 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Conversation prompt templates.
3
+
4
+ We kindly request that you import fastchat instead of copying this file if you wish to use it.
5
+ If you have changes in mind, please contribute back so the community can benefit collectively and continue to maintain these valuable templates.
6
+ """
7
+
8
+ import dataclasses
9
+ from enum import IntEnum, auto
10
+ from typing import Any, Dict, List, Tuple, Union
11
+
12
+
13
+ class SeparatorStyle(IntEnum):
14
+ """Separator styles."""
15
+
16
+ ADD_COLON_SINGLE = auto()
17
+ ADD_COLON_TWO = auto()
18
+ ADD_COLON_SPACE_SINGLE = auto()
19
+ NO_COLON_SINGLE = auto()
20
+ NO_COLON_TWO = auto()
21
+ ADD_NEW_LINE_SINGLE = auto()
22
+ LLAMA2 = auto()
23
+ CHATGLM = auto()
24
+ CHATML = auto()
25
+ CHATINTERN = auto()
26
+ DOLLY = auto()
27
+ RWKV = auto()
28
+ PHOENIX = auto()
29
+ ROBIN = auto()
30
+ FALCON_CHAT = auto()
31
+ CHATGLM3 = auto()
32
+ INTERNVL_ZH = auto()
33
+ MPT = auto()
34
+
35
+
36
+ @dataclasses.dataclass
37
+ class Conversation:
38
+ """A class that manages prompt templates and keeps all conversation history."""
39
+
40
+ # The name of this template
41
+ name: str
42
+ # The template of the system prompt
43
+ system_template: str = '{system_message}'
44
+ # The system message
45
+ system_message: str = ''
46
+ # The names of two roles
47
+ roles: Tuple[str] = ('USER', 'ASSISTANT')
48
+ # All messages. Each item is (role, message).
49
+ messages: List[List[str]] = ()
50
+ # The number of few shot examples
51
+ offset: int = 0
52
+ # The separator style and configurations
53
+ sep_style: SeparatorStyle = SeparatorStyle.ADD_COLON_SINGLE
54
+ sep: str = '\n'
55
+ sep2: str = None
56
+ # Stop criteria (the default one is EOS token)
57
+ stop_str: Union[str, List[str]] = None
58
+ # Stops generation if meeting any token in this list
59
+ stop_token_ids: List[int] = None
60
+
61
+ def get_prompt(self) -> str:
62
+ """Get the prompt for generation."""
63
+ system_prompt = self.system_template.format(system_message=self.system_message)
64
+ if self.sep_style == SeparatorStyle.ADD_COLON_SINGLE:
65
+ ret = system_prompt + self.sep
66
+ for role, message in self.messages:
67
+ if message:
68
+ ret += role + ': ' + message + self.sep
69
+ else:
70
+ ret += role + ':'
71
+ return ret
72
+ elif self.sep_style == SeparatorStyle.ADD_COLON_TWO:
73
+ seps = [self.sep, self.sep2]
74
+ ret = system_prompt + seps[0]
75
+ for i, (role, message) in enumerate(self.messages):
76
+ if message:
77
+ ret += role + ': ' + message + seps[i % 2]
78
+ else:
79
+ ret += role + ':'
80
+ return ret
81
+ elif self.sep_style == SeparatorStyle.ADD_COLON_SPACE_SINGLE:
82
+ ret = system_prompt + self.sep
83
+ for role, message in self.messages:
84
+ if message:
85
+ ret += role + ': ' + message + self.sep
86
+ else:
87
+ ret += role + ': ' # must be end with a space
88
+ return ret
89
+ elif self.sep_style == SeparatorStyle.ADD_NEW_LINE_SINGLE:
90
+ ret = '' if system_prompt == '' else system_prompt + self.sep
91
+ for role, message in self.messages:
92
+ if message:
93
+ ret += role + '\n' + message + self.sep
94
+ else:
95
+ ret += role + '\n'
96
+ return ret
97
+ elif self.sep_style == SeparatorStyle.NO_COLON_SINGLE:
98
+ ret = system_prompt
99
+ for role, message in self.messages:
100
+ if message:
101
+ ret += role + message + self.sep
102
+ else:
103
+ ret += role
104
+ return ret
105
+ elif self.sep_style == SeparatorStyle.NO_COLON_TWO:
106
+ seps = [self.sep, self.sep2]
107
+ ret = system_prompt
108
+ for i, (role, message) in enumerate(self.messages):
109
+ if message:
110
+ ret += role + message + seps[i % 2]
111
+ else:
112
+ ret += role
113
+ return ret
114
+ elif self.sep_style == SeparatorStyle.RWKV:
115
+ ret = system_prompt
116
+ for i, (role, message) in enumerate(self.messages):
117
+ if message:
118
+ ret += (
119
+ role
120
+ + ': '
121
+ + message.replace('\r\n', '\n').replace('\n\n', '\n')
122
+ )
123
+ ret += '\n\n'
124
+ else:
125
+ ret += role + ':'
126
+ return ret
127
+ elif self.sep_style == SeparatorStyle.LLAMA2:
128
+ seps = [self.sep, self.sep2]
129
+ if self.system_message:
130
+ ret = system_prompt
131
+ else:
132
+ ret = '[INST] '
133
+ for i, (role, message) in enumerate(self.messages):
134
+ tag = self.roles[i % 2]
135
+ if message:
136
+ if i == 0:
137
+ ret += message + ' '
138
+ else:
139
+ ret += tag + ' ' + message + seps[i % 2]
140
+ else:
141
+ ret += tag
142
+ return ret
143
+ elif self.sep_style == SeparatorStyle.CHATGLM:
144
+ # source: https://huggingface.co/THUDM/chatglm-6b/blob/1d240ba371910e9282298d4592532d7f0f3e9f3e/modeling_chatglm.py#L1302-L1308
145
+ # source2: https://huggingface.co/THUDM/chatglm2-6b/blob/e186c891cf64310ac66ef10a87e6635fa6c2a579/modeling_chatglm.py#L926
146
+ round_add_n = 1 if self.name == 'chatglm2' else 0
147
+ if system_prompt:
148
+ ret = system_prompt + self.sep
149
+ else:
150
+ ret = ''
151
+
152
+ for i, (role, message) in enumerate(self.messages):
153
+ if i % 2 == 0:
154
+ ret += f'[Round {i//2 + round_add_n}]{self.sep}'
155
+
156
+ if message:
157
+ ret += f'{role}:{message}{self.sep}'
158
+ else:
159
+ ret += f'{role}:'
160
+ return ret
161
+ elif self.sep_style == SeparatorStyle.CHATML:
162
+ ret = '' if system_prompt == '' else system_prompt + self.sep + '\n'
163
+ for role, message in self.messages:
164
+ if message:
165
+ ret += role + '\n' + message + self.sep + '\n'
166
+ else:
167
+ ret += role + '\n'
168
+ return ret
169
+ elif self.sep_style == SeparatorStyle.CHATGLM3:
170
+ ret = ''
171
+ if self.system_message:
172
+ ret += system_prompt
173
+ for role, message in self.messages:
174
+ if message:
175
+ ret += role + '\n' + ' ' + message
176
+ else:
177
+ ret += role
178
+ return ret
179
+ elif self.sep_style == SeparatorStyle.CHATINTERN:
180
+ # source: https://huggingface.co/internlm/internlm-chat-7b-8k/blob/bd546fa984b4b0b86958f56bf37f94aa75ab8831/modeling_internlm.py#L771
181
+ seps = [self.sep, self.sep2]
182
+ ret = system_prompt
183
+ for i, (role, message) in enumerate(self.messages):
184
+ # if i % 2 == 0:
185
+ # ret += "<s>"
186
+ if message:
187
+ ret += role + ':' + message + seps[i % 2] + '\n'
188
+ else:
189
+ ret += role + ':'
190
+ return ret
191
+ elif self.sep_style == SeparatorStyle.DOLLY:
192
+ seps = [self.sep, self.sep2]
193
+ ret = system_prompt
194
+ for i, (role, message) in enumerate(self.messages):
195
+ if message:
196
+ ret += role + ':\n' + message + seps[i % 2]
197
+ if i % 2 == 1:
198
+ ret += '\n\n'
199
+ else:
200
+ ret += role + ':\n'
201
+ return ret
202
+ elif self.sep_style == SeparatorStyle.PHOENIX:
203
+ ret = system_prompt
204
+ for role, message in self.messages:
205
+ if message:
206
+ ret += role + ': ' + '<s>' + message + '</s>'
207
+ else:
208
+ ret += role + ': ' + '<s>'
209
+ return ret
210
+ elif self.sep_style == SeparatorStyle.ROBIN:
211
+ ret = system_prompt + self.sep
212
+ for role, message in self.messages:
213
+ if message:
214
+ ret += role + ':\n' + message + self.sep
215
+ else:
216
+ ret += role + ':\n'
217
+ return ret
218
+ elif self.sep_style == SeparatorStyle.FALCON_CHAT:
219
+ ret = ''
220
+ if self.system_message:
221
+ ret += system_prompt + self.sep
222
+ for role, message in self.messages:
223
+ if message:
224
+ ret += role + ': ' + message + self.sep
225
+ else:
226
+ ret += role + ':'
227
+
228
+ return ret
229
+ elif self.sep_style == SeparatorStyle.INTERNVL_ZH:
230
+ seps = [self.sep, self.sep2]
231
+ ret = self.system_message + seps[0]
232
+ for i, (role, message) in enumerate(self.messages):
233
+ if message:
234
+ ret += role + ': ' + message + seps[i % 2]
235
+ else:
236
+ ret += role + ':'
237
+ return ret
238
+ elif self.sep_style == SeparatorStyle.MPT:
239
+ ret = system_prompt + self.sep
240
+ for role, message in self.messages:
241
+ if message:
242
+ if type(message) is tuple:
243
+ message, _, _ = message
244
+ ret += role + message + self.sep
245
+ else:
246
+ ret += role
247
+ return ret
248
+ else:
249
+ raise ValueError(f'Invalid style: {self.sep_style}')
250
+
251
+ def set_system_message(self, system_message: str):
252
+ """Set the system message."""
253
+ self.system_message = system_message
254
+
255
+ def append_message(self, role: str, message: str):
256
+ """Append a new message."""
257
+ self.messages.append([role, message])
258
+
259
+ def update_last_message(self, message: str):
260
+ """Update the last output.
261
+
262
+ The last message is typically set to be None when constructing the prompt,
263
+ so we need to update it in-place after getting the response from a model.
264
+ """
265
+ self.messages[-1][1] = message
266
+
267
+ def to_gradio_chatbot(self):
268
+ """Convert the conversation to gradio chatbot format."""
269
+ ret = []
270
+ for i, (role, msg) in enumerate(self.messages[self.offset :]):
271
+ if i % 2 == 0:
272
+ ret.append([msg, None])
273
+ else:
274
+ ret[-1][-1] = msg
275
+ return ret
276
+
277
+ def to_openai_api_messages(self):
278
+ """Convert the conversation to OpenAI chat completion format."""
279
+ ret = [{'role': 'system', 'content': self.system_message}]
280
+
281
+ for i, (_, msg) in enumerate(self.messages[self.offset :]):
282
+ if i % 2 == 0:
283
+ ret.append({'role': 'user', 'content': msg})
284
+ else:
285
+ if msg is not None:
286
+ ret.append({'role': 'assistant', 'content': msg})
287
+ return ret
288
+
289
+ def copy(self):
290
+ return Conversation(
291
+ name=self.name,
292
+ system_template=self.system_template,
293
+ system_message=self.system_message,
294
+ roles=self.roles,
295
+ messages=[[x, y] for x, y in self.messages],
296
+ offset=self.offset,
297
+ sep_style=self.sep_style,
298
+ sep=self.sep,
299
+ sep2=self.sep2,
300
+ stop_str=self.stop_str,
301
+ stop_token_ids=self.stop_token_ids,
302
+ )
303
+
304
+ def dict(self):
305
+ return {
306
+ 'template_name': self.name,
307
+ 'system_message': self.system_message,
308
+ 'roles': self.roles,
309
+ 'messages': self.messages,
310
+ 'offset': self.offset,
311
+ }
312
+
313
+
314
+ # A global registry for all conversation templates
315
+ conv_templates: Dict[str, Conversation] = {}
316
+
317
+
318
+ def register_conv_template(template: Conversation, override: bool = False):
319
+ """Register a new conversation template."""
320
+ if not override:
321
+ assert (
322
+ template.name not in conv_templates
323
+ ), f'{template.name} has been registered.'
324
+
325
+ conv_templates[template.name] = template
326
+
327
+
328
+ def get_conv_template(name: str) -> Conversation:
329
+ """Get a conversation template."""
330
+ return conv_templates[name].copy()
331
+
332
+
333
+ # Note that for inference, using the Hermes-2 and internlm2-chat templates is equivalent.
334
+ register_conv_template(
335
+ Conversation(
336
+ name='Hermes-2',
337
+ system_template='<|im_start|>system\n{system_message}',
338
+ # note: The new system prompt was not used here to avoid changes in benchmark performance.
339
+ # system_message='我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。',
340
+ system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型,英文名叫InternVL, 是一个有用无害的人工智能助手。',
341
+ roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
342
+ sep_style=SeparatorStyle.MPT,
343
+ sep='<|im_end|>',
344
+ stop_token_ids=[
345
+ 2,
346
+ 6,
347
+ 7,
348
+ 8,
349
+ ],
350
+ stop_str='<|endoftext|>',
351
+ )
352
+ )
353
+
354
+
355
+ register_conv_template(
356
+ Conversation(
357
+ name='internlm2-chat',
358
+ system_template='<|im_start|>system\n{system_message}',
359
+ # note: The new system prompt was not used here to avoid changes in benchmark performance.
360
+ # system_message='我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。',
361
+ system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型,英文名叫InternVL, 是一个有用无害的人工智能助手。',
362
+ roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
363
+ sep_style=SeparatorStyle.MPT,
364
+ sep='<|im_end|>',
365
+ stop_token_ids=[
366
+ 2,
367
+ 92543,
368
+ 92542
369
+ ]
370
+ )
371
+ )
372
+
373
+
374
+ register_conv_template(
375
+ Conversation(
376
+ name='phi3-chat',
377
+ system_template='<|system|>\n{system_message}',
378
+ # note: The new system prompt was not used here to avoid changes in benchmark performance.
379
+ # system_message='我是书生·万象,英文名是InternVL,是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新,开源开放,共享共创,推动科技进步和产业发展。',
380
+ system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型,英文名叫InternVL, 是一个有用无害的人工智能助手。',
381
+ roles=('<|user|>\n', '<|assistant|>\n'),
382
+ sep_style=SeparatorStyle.MPT,
383
+ sep='<|end|>',
384
+ stop_token_ids=[
385
+ 2,
386
+ 32000,
387
+ 32007
388
+ ]
389
+ )
390
+ )
examples/image1.jpg ADDED
examples/image2.jpg ADDED
examples/red-panda.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d921c07bb97224d65a37801541d246067f0d506f08723ffa1ad85c217907ccb8
3
+ size 1867237
generation_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "transformers_version": "4.37.2"
4
+ }
model-00001-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2e9c8f2eef93cca26b264644bbd1e60628509ba86cbd165d7dfcefae31a145c
3
+ size 4988569440
model-00002-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c9eb45570a3d8ef7531b3b3639aeee3332306743e6f6fc7ea1016df2dea6e71
3
+ size 4937253584
model-00003-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52dd534aad2a47b0b090095503ec0919c3cfeb5d06e605a16eb34ef519a76ab9
3
+ size 4824755816
model-00004-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:df877c2baafa117b17f01a95d138871b94f9e96fc7aaddf9a285cdd510c4d6ec
3
+ size 4756460272
model-00005-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:780130f0ee53e957929fcb9464aa9a589fc3b679deedba18597af1ccab790e83
3
+ size 4991370776
model-00006-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b14d64e3fa3df206e0d1c9a1fa9e580cd0b5c353eee68a7d5e54bf0c78b4b32f
3
+ size 4756460312
model-00007-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f618c5331913ced2396d4f12680da1b56ca2b55a88c3a2c95829ecf8e3bfb4f
3
+ size 4756460312
model-00008-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:956f796b54b3ae052b45b9782c1d9105826d4de52edda10f858d23d460073f22
3
+ size 4991370808
model-00009-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83f3ba456f52aece1995dc0189a2ea076fb0cae34bafa9c0645422181ce8f9de
3
+ size 4756460312
model-00010-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:681d56b93f433bddf6c852588c6434c9b4ef98d84ec52dc874b9ce4018eddc68
3
+ size 4756460312
model-00011-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1b6777cb681d25f05fe6eee1ff9b8e1616b364564a1ea94863f63c32c5ae56f0
3
+ size 4991370808
model-00012-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:603ebafa77744de964ad013bb10e01336c9fb6b077f917800040b0ef177aa2bd
3
+ size 4756460312
model-00013-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b37c1abdc6235f39ac3f1c543da51cec39b862daea98ca552f36fd1e69ab747f
3
+ size 4756460312
model-00014-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f667ade7d23d434b8629043085a4b520a66fcee28fb02786d927cf15e567f51
3
+ size 4991370808
model-00015-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe21d55429ad26c3e2f4bc5c0a9f3acd2975a99417f9f7805c1740231f1482d8
3
+ size 4756460312
model-00016-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae329cb9495eb32b8bf3bbfae93925538ab3e2fd4190e4f1f2fed4dd28091d64
3
+ size 4756460312
model-00017-of-00017.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:16ebf24219a66f553f3e5742a4525998c14adc3466bad489828637e97ad8730b
3
+ size 2613305648
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
modeling_intern_vit.py ADDED
@@ -0,0 +1,434 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # InternVL
3
+ # Copyright (c) 2023 OpenGVLab
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # --------------------------------------------------------
6
+ from typing import Optional, Tuple, Union
7
+
8
+ import torch
9
+ import torch.nn.functional as F
10
+ import torch.utils.checkpoint
11
+ from einops import rearrange
12
+ from timm.models.layers import DropPath
13
+ from torch import nn
14
+ from transformers.activations import ACT2FN
15
+ from transformers.modeling_outputs import (BaseModelOutput,
16
+ BaseModelOutputWithPooling)
17
+ from transformers.modeling_utils import PreTrainedModel
18
+ from transformers.utils import logging
19
+
20
+ from .configuration_intern_vit import InternVisionConfig
21
+
22
+ try:
23
+ try: # v1
24
+ from flash_attn.flash_attn_interface import \
25
+ flash_attn_unpadded_qkvpacked_func
26
+ except: # v2
27
+ from flash_attn.flash_attn_interface import \
28
+ flash_attn_varlen_qkvpacked_func as flash_attn_unpadded_qkvpacked_func
29
+
30
+ from flash_attn.bert_padding import pad_input, unpad_input
31
+
32
+ has_flash_attn = True
33
+ except:
34
+ print('FlashAttention is not installed.')
35
+ has_flash_attn = False
36
+
37
+ logger = logging.get_logger(__name__)
38
+
39
+
40
+ class FlashAttention(nn.Module):
41
+ """Implement the scaled dot product attention with softmax.
42
+ Arguments
43
+ ---------
44
+ softmax_scale: The temperature to use for the softmax attention.
45
+ (default: 1/sqrt(d_keys) where d_keys is computed at
46
+ runtime)
47
+ attention_dropout: The dropout rate to apply to the attention
48
+ (default: 0.0)
49
+ """
50
+
51
+ def __init__(self, softmax_scale=None, attention_dropout=0.0, device=None, dtype=None):
52
+ super().__init__()
53
+ self.softmax_scale = softmax_scale
54
+ self.dropout_p = attention_dropout
55
+
56
+ def forward(self, qkv, key_padding_mask=None, causal=False, cu_seqlens=None,
57
+ max_s=None, need_weights=False):
58
+ """Implements the multihead softmax attention.
59
+ Arguments
60
+ ---------
61
+ qkv: The tensor containing the query, key, and value. (B, S, 3, H, D) if key_padding_mask is None
62
+ if unpadded: (nnz, 3, h, d)
63
+ key_padding_mask: a bool tensor of shape (B, S)
64
+ """
65
+ assert not need_weights
66
+ assert qkv.dtype in [torch.float16, torch.bfloat16]
67
+ assert qkv.is_cuda
68
+
69
+ if cu_seqlens is None:
70
+ batch_size = qkv.shape[0]
71
+ seqlen = qkv.shape[1]
72
+ if key_padding_mask is None:
73
+ qkv = rearrange(qkv, 'b s ... -> (b s) ...')
74
+ max_s = seqlen
75
+ cu_seqlens = torch.arange(0, (batch_size + 1) * seqlen, step=seqlen, dtype=torch.int32,
76
+ device=qkv.device)
77
+ output = flash_attn_unpadded_qkvpacked_func(
78
+ qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
79
+ softmax_scale=self.softmax_scale, causal=causal
80
+ )
81
+ output = rearrange(output, '(b s) ... -> b s ...', b=batch_size)
82
+ else:
83
+ nheads = qkv.shape[-2]
84
+ x = rearrange(qkv, 'b s three h d -> b s (three h d)')
85
+ x_unpad, indices, cu_seqlens, max_s = unpad_input(x, key_padding_mask)
86
+ x_unpad = rearrange(x_unpad, 'nnz (three h d) -> nnz three h d', three=3, h=nheads)
87
+ output_unpad = flash_attn_unpadded_qkvpacked_func(
88
+ x_unpad, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
89
+ softmax_scale=self.softmax_scale, causal=causal
90
+ )
91
+ output = rearrange(pad_input(rearrange(output_unpad, 'nnz h d -> nnz (h d)'),
92
+ indices, batch_size, seqlen),
93
+ 'b s (h d) -> b s h d', h=nheads)
94
+ else:
95
+ assert max_s is not None
96
+ output = flash_attn_unpadded_qkvpacked_func(
97
+ qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
98
+ softmax_scale=self.softmax_scale, causal=causal
99
+ )
100
+
101
+ return output, None
102
+
103
+
104
+ class InternRMSNorm(nn.Module):
105
+ def __init__(self, hidden_size, eps=1e-6):
106
+ super().__init__()
107
+ self.weight = nn.Parameter(torch.ones(hidden_size))
108
+ self.variance_epsilon = eps
109
+
110
+ def forward(self, hidden_states):
111
+ input_dtype = hidden_states.dtype
112
+ hidden_states = hidden_states.to(torch.float32)
113
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
114
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
115
+ return self.weight * hidden_states.to(input_dtype)
116
+
117
+
118
+ try:
119
+ from apex.normalization import FusedRMSNorm
120
+
121
+ InternRMSNorm = FusedRMSNorm # noqa
122
+
123
+ logger.info('Discovered apex.normalization.FusedRMSNorm - will use it instead of InternRMSNorm')
124
+ except ImportError:
125
+ # using the normal InternRMSNorm
126
+ pass
127
+ except Exception:
128
+ logger.warning('discovered apex but it failed to load, falling back to InternRMSNorm')
129
+ pass
130
+
131
+
132
+ NORM2FN = {
133
+ 'rms_norm': InternRMSNorm,
134
+ 'layer_norm': nn.LayerNorm,
135
+ }
136
+
137
+
138
+ class InternVisionEmbeddings(nn.Module):
139
+ def __init__(self, config: InternVisionConfig):
140
+ super().__init__()
141
+ self.config = config
142
+ self.embed_dim = config.hidden_size
143
+ self.image_size = config.image_size
144
+ self.patch_size = config.patch_size
145
+
146
+ self.class_embedding = nn.Parameter(
147
+ torch.randn(1, 1, self.embed_dim),
148
+ )
149
+
150
+ self.patch_embedding = nn.Conv2d(
151
+ in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size
152
+ )
153
+
154
+ self.num_patches = (self.image_size // self.patch_size) ** 2
155
+ self.num_positions = self.num_patches + 1
156
+
157
+ self.position_embedding = nn.Parameter(torch.randn(1, self.num_positions, self.embed_dim))
158
+
159
+ def _get_pos_embed(self, pos_embed, H, W):
160
+ target_dtype = pos_embed.dtype
161
+ pos_embed = pos_embed.float().reshape(
162
+ 1, self.image_size // self.patch_size, self.image_size // self.patch_size, -1).permute(0, 3, 1, 2)
163
+ pos_embed = F.interpolate(pos_embed, size=(H, W), mode='bicubic', align_corners=False). \
164
+ reshape(1, -1, H * W).permute(0, 2, 1).to(target_dtype)
165
+ return pos_embed
166
+
167
+ def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor:
168
+ target_dtype = self.patch_embedding.weight.dtype
169
+ patch_embeds = self.patch_embedding(pixel_values) # shape = [*, channel, width, height]
170
+ batch_size, _, height, width = patch_embeds.shape
171
+ patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
172
+ class_embeds = self.class_embedding.expand(batch_size, 1, -1).to(target_dtype)
173
+ embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
174
+ position_embedding = torch.cat([
175
+ self.position_embedding[:, :1, :],
176
+ self._get_pos_embed(self.position_embedding[:, 1:, :], height, width)
177
+ ], dim=1)
178
+ embeddings = embeddings + position_embedding.to(target_dtype)
179
+ return embeddings
180
+
181
+
182
+ class InternAttention(nn.Module):
183
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
184
+
185
+ def __init__(self, config: InternVisionConfig):
186
+ super().__init__()
187
+ self.config = config
188
+ self.embed_dim = config.hidden_size
189
+ self.num_heads = config.num_attention_heads
190
+ self.use_flash_attn = config.use_flash_attn and has_flash_attn
191
+ if config.use_flash_attn and not has_flash_attn:
192
+ print('Warning: Flash Attention is not available, use_flash_attn is set to False.')
193
+ self.head_dim = self.embed_dim // self.num_heads
194
+ if self.head_dim * self.num_heads != self.embed_dim:
195
+ raise ValueError(
196
+ f'embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:'
197
+ f' {self.num_heads}).'
198
+ )
199
+
200
+ self.scale = self.head_dim ** -0.5
201
+ self.qkv = nn.Linear(self.embed_dim, 3 * self.embed_dim, bias=config.qkv_bias)
202
+ self.attn_drop = nn.Dropout(config.attention_dropout)
203
+ self.proj_drop = nn.Dropout(config.dropout)
204
+
205
+ self.qk_normalization = config.qk_normalization
206
+
207
+ if self.qk_normalization:
208
+ self.q_norm = InternRMSNorm(self.embed_dim, eps=config.layer_norm_eps)
209
+ self.k_norm = InternRMSNorm(self.embed_dim, eps=config.layer_norm_eps)
210
+
211
+ if self.use_flash_attn:
212
+ self.inner_attn = FlashAttention(attention_dropout=config.attention_dropout)
213
+ self.proj = nn.Linear(self.embed_dim, self.embed_dim)
214
+
215
+ def _naive_attn(self, x):
216
+ B, N, C = x.shape
217
+ qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
218
+ q, k, v = qkv.unbind(0) # make torchscript happy (cannot use tensor as tuple)
219
+
220
+ if self.qk_normalization:
221
+ B_, H_, N_, D_ = q.shape
222
+ q = self.q_norm(q.transpose(1, 2).flatten(-2, -1)).view(B_, N_, H_, D_).transpose(1, 2)
223
+ k = self.k_norm(k.transpose(1, 2).flatten(-2, -1)).view(B_, N_, H_, D_).transpose(1, 2)
224
+
225
+ attn = ((q * self.scale) @ k.transpose(-2, -1))
226
+ attn = attn.softmax(dim=-1)
227
+ attn = self.attn_drop(attn)
228
+
229
+ x = (attn @ v).transpose(1, 2).reshape(B, N, C)
230
+ x = self.proj(x)
231
+ x = self.proj_drop(x)
232
+ return x
233
+
234
+ def _flash_attn(self, x, key_padding_mask=None, need_weights=False):
235
+ qkv = self.qkv(x)
236
+ qkv = rearrange(qkv, 'b s (three h d) -> b s three h d', three=3, h=self.num_heads)
237
+
238
+ if self.qk_normalization:
239
+ q, k, v = qkv.unbind(2)
240
+ q = self.q_norm(q.flatten(-2, -1)).view(q.shape)
241
+ k = self.k_norm(k.flatten(-2, -1)).view(k.shape)
242
+ qkv = torch.stack([q, k, v], dim=2)
243
+
244
+ context, _ = self.inner_attn(
245
+ qkv, key_padding_mask=key_padding_mask, need_weights=need_weights, causal=False
246
+ )
247
+ outs = self.proj(rearrange(context, 'b s h d -> b s (h d)'))
248
+ outs = self.proj_drop(outs)
249
+ return outs
250
+
251
+ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
252
+ x = self._naive_attn(hidden_states) if not self.use_flash_attn else self._flash_attn(hidden_states)
253
+ return x
254
+
255
+
256
+ class InternMLP(nn.Module):
257
+ def __init__(self, config: InternVisionConfig):
258
+ super().__init__()
259
+ self.config = config
260
+ self.act = ACT2FN[config.hidden_act]
261
+ self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
262
+ self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
263
+
264
+ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
265
+ hidden_states = self.fc1(hidden_states)
266
+ hidden_states = self.act(hidden_states)
267
+ hidden_states = self.fc2(hidden_states)
268
+ return hidden_states
269
+
270
+
271
+ class InternVisionEncoderLayer(nn.Module):
272
+ def __init__(self, config: InternVisionConfig, drop_path_rate: float):
273
+ super().__init__()
274
+ self.embed_dim = config.hidden_size
275
+ self.intermediate_size = config.intermediate_size
276
+ self.norm_type = config.norm_type
277
+
278
+ self.attn = InternAttention(config)
279
+ self.mlp = InternMLP(config)
280
+ self.norm1 = NORM2FN[self.norm_type](self.embed_dim, eps=config.layer_norm_eps)
281
+ self.norm2 = NORM2FN[self.norm_type](self.embed_dim, eps=config.layer_norm_eps)
282
+
283
+ self.ls1 = nn.Parameter(config.initializer_factor * torch.ones(self.embed_dim))
284
+ self.ls2 = nn.Parameter(config.initializer_factor * torch.ones(self.embed_dim))
285
+ self.drop_path1 = DropPath(drop_path_rate) if drop_path_rate > 0. else nn.Identity()
286
+ self.drop_path2 = DropPath(drop_path_rate) if drop_path_rate > 0. else nn.Identity()
287
+
288
+ def forward(
289
+ self,
290
+ hidden_states: torch.Tensor,
291
+ ) -> Tuple[torch.FloatTensor, Optional[torch.FloatTensor], Optional[Tuple[torch.FloatTensor]]]:
292
+ """
293
+ Args:
294
+ hidden_states (`Tuple[torch.FloatTensor, Optional[torch.FloatTensor]]`): input to the layer of shape `(batch, seq_len, embed_dim)`
295
+ """
296
+ hidden_states = hidden_states + self.drop_path1(self.attn(self.norm1(hidden_states)) * self.ls1)
297
+
298
+ hidden_states = hidden_states + self.drop_path2(self.mlp(self.norm2(hidden_states)) * self.ls2)
299
+
300
+ return hidden_states
301
+
302
+
303
+ class InternVisionEncoder(nn.Module):
304
+ """
305
+ Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a
306
+ [`InternEncoderLayer`].
307
+
308
+ Args:
309
+ config (`InternConfig`):
310
+ The corresponding vision configuration for the `InternEncoder`.
311
+ """
312
+
313
+ def __init__(self, config: InternVisionConfig):
314
+ super().__init__()
315
+ self.config = config
316
+ # stochastic depth decay rule
317
+ dpr = [x.item() for x in torch.linspace(0, config.drop_path_rate, config.num_hidden_layers)]
318
+ self.layers = nn.ModuleList([
319
+ InternVisionEncoderLayer(config, dpr[idx]) for idx in range(config.num_hidden_layers)])
320
+ self.gradient_checkpointing = True
321
+
322
+ def forward(
323
+ self,
324
+ inputs_embeds,
325
+ output_hidden_states: Optional[bool] = None,
326
+ return_dict: Optional[bool] = None,
327
+ ) -> Union[Tuple, BaseModelOutput]:
328
+ r"""
329
+ Args:
330
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
331
+ Embedded representation of the inputs. Should be float, not int tokens.
332
+ output_hidden_states (`bool`, *optional*):
333
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
334
+ for more detail.
335
+ return_dict (`bool`, *optional*):
336
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
337
+ """
338
+ output_hidden_states = (
339
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
340
+ )
341
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
342
+
343
+ encoder_states = () if output_hidden_states else None
344
+ hidden_states = inputs_embeds
345
+
346
+ for idx, encoder_layer in enumerate(self.layers):
347
+ if output_hidden_states:
348
+ encoder_states = encoder_states + (hidden_states,)
349
+ if self.gradient_checkpointing and self.training:
350
+ layer_outputs = torch.utils.checkpoint.checkpoint(
351
+ encoder_layer,
352
+ hidden_states)
353
+ else:
354
+ layer_outputs = encoder_layer(
355
+ hidden_states,
356
+ )
357
+ hidden_states = layer_outputs
358
+
359
+ if output_hidden_states:
360
+ encoder_states = encoder_states + (hidden_states,)
361
+
362
+ if not return_dict:
363
+ return tuple(v for v in [hidden_states, encoder_states] if v is not None)
364
+ return BaseModelOutput(
365
+ last_hidden_state=hidden_states, hidden_states=encoder_states
366
+ )
367
+
368
+
369
+ class InternVisionModel(PreTrainedModel):
370
+ main_input_name = 'pixel_values'
371
+ config_class = InternVisionConfig
372
+ _no_split_modules = ['InternVisionEncoderLayer']
373
+
374
+ def __init__(self, config: InternVisionConfig):
375
+ super().__init__(config)
376
+ self.config = config
377
+
378
+ self.embeddings = InternVisionEmbeddings(config)
379
+ self.encoder = InternVisionEncoder(config)
380
+
381
+ def resize_pos_embeddings(self, old_size, new_size, patch_size):
382
+ pos_emb = self.embeddings.position_embedding
383
+ _, num_positions, embed_dim = pos_emb.shape
384
+ cls_emb = pos_emb[:, :1, :]
385
+ pos_emb = pos_emb[:, 1:, :].reshape(1, old_size // patch_size, old_size // patch_size, -1).permute(0, 3, 1, 2)
386
+ pos_emb = F.interpolate(pos_emb.float(), size=new_size // patch_size, mode='bicubic', align_corners=False)
387
+ pos_emb = pos_emb.to(cls_emb.dtype).reshape(1, embed_dim, -1).permute(0, 2, 1)
388
+ pos_emb = torch.cat([cls_emb, pos_emb], dim=1)
389
+ self.embeddings.position_embedding = nn.Parameter(pos_emb)
390
+ self.embeddings.image_size = new_size
391
+ logger.info('Resized position embeddings from {} to {}'.format(old_size, new_size))
392
+
393
+ def get_input_embeddings(self):
394
+ return self.embeddings
395
+
396
+ def forward(
397
+ self,
398
+ pixel_values: Optional[torch.FloatTensor] = None,
399
+ output_hidden_states: Optional[bool] = None,
400
+ return_dict: Optional[bool] = None,
401
+ pixel_embeds: Optional[torch.FloatTensor] = None,
402
+ ) -> Union[Tuple, BaseModelOutputWithPooling]:
403
+ output_hidden_states = (
404
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
405
+ )
406
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
407
+
408
+ if pixel_values is None and pixel_embeds is None:
409
+ raise ValueError('You have to specify pixel_values or pixel_embeds')
410
+
411
+ if pixel_embeds is not None:
412
+ hidden_states = pixel_embeds
413
+ else:
414
+ if len(pixel_values.shape) == 4:
415
+ hidden_states = self.embeddings(pixel_values)
416
+ else:
417
+ raise ValueError(f'wrong pixel_values size: {pixel_values.shape}')
418
+ encoder_outputs = self.encoder(
419
+ inputs_embeds=hidden_states,
420
+ output_hidden_states=output_hidden_states,
421
+ return_dict=return_dict,
422
+ )
423
+ last_hidden_state = encoder_outputs.last_hidden_state
424
+ pooled_output = last_hidden_state[:, 0, :]
425
+
426
+ if not return_dict:
427
+ return (last_hidden_state, pooled_output) + encoder_outputs[1:]
428
+
429
+ return BaseModelOutputWithPooling(
430
+ last_hidden_state=last_hidden_state,
431
+ pooler_output=pooled_output,
432
+ hidden_states=encoder_outputs.hidden_states,
433
+ attentions=encoder_outputs.attentions,
434
+ )
modeling_internvl_chat.py ADDED
@@ -0,0 +1,337 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # InternVL
3
+ # Copyright (c) 2024 OpenGVLab
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # --------------------------------------------------------
6
+ import warnings
7
+ from typing import Any, List, Optional, Tuple, Union
8
+
9
+ import torch.utils.checkpoint
10
+ import transformers
11
+ from torch import nn
12
+ from torch.nn import CrossEntropyLoss
13
+ from transformers import AutoModel, GenerationConfig, LlamaForCausalLM
14
+ from transformers.modeling_outputs import CausalLMOutputWithPast
15
+ from transformers.modeling_utils import PreTrainedModel
16
+ from transformers.utils import ModelOutput, logging
17
+
18
+ from .configuration_internvl_chat import InternVLChatConfig
19
+ from .conversation import get_conv_template
20
+ from .modeling_intern_vit import InternVisionModel
21
+
22
+ logger = logging.get_logger(__name__)
23
+
24
+
25
+ def version_cmp(v1, v2, op='eq'):
26
+ import operator
27
+
28
+ from packaging import version
29
+ op_func = getattr(operator, op)
30
+ return op_func(version.parse(v1), version.parse(v2))
31
+
32
+
33
+ class InternVLChatModel(PreTrainedModel):
34
+ config_class = InternVLChatConfig
35
+ main_input_name = 'pixel_values'
36
+ _no_split_modules = ['InternVisionModel', 'LlamaDecoderLayer']
37
+
38
+ def __init__(self, config: InternVLChatConfig, vision_model=None, language_model=None):
39
+ super().__init__(config)
40
+
41
+ assert version_cmp(transformers.__version__, '4.36.2', 'ge')
42
+ image_size = config.force_image_size or config.vision_config.image_size
43
+ patch_size = config.vision_config.patch_size
44
+ self.patch_size = patch_size
45
+ self.select_layer = config.select_layer
46
+ self.template = config.template
47
+ self.num_image_token = int((image_size // patch_size) ** 2 * (config.downsample_ratio ** 2))
48
+ self.downsample_ratio = config.downsample_ratio
49
+ self.ps_version = config.ps_version
50
+
51
+ logger.info(f'num_image_token: {self.num_image_token}')
52
+ logger.info(f'ps_version: {self.ps_version}')
53
+ if vision_model is not None:
54
+ self.vision_model = vision_model
55
+ else:
56
+ self.vision_model = InternVisionModel(config.vision_config)
57
+ if language_model is not None:
58
+ self.language_model = language_model
59
+ else:
60
+ if config.llm_config.architectures[0] == 'LlamaForCausalLM':
61
+ self.language_model = LlamaForCausalLM(config.llm_config)
62
+ else:
63
+ raise NotImplementedError(f'{config.llm_config.architectures[0]} is not implemented.')
64
+
65
+ vit_hidden_size = config.vision_config.hidden_size
66
+ llm_hidden_size = config.llm_config.hidden_size
67
+
68
+ self.mlp1 = nn.Sequential(
69
+ nn.LayerNorm(vit_hidden_size * int(1 / self.downsample_ratio) ** 2),
70
+ nn.Linear(vit_hidden_size * int(1 / self.downsample_ratio) ** 2, llm_hidden_size),
71
+ nn.GELU(),
72
+ nn.Linear(llm_hidden_size, llm_hidden_size)
73
+ )
74
+
75
+ self.img_context_token_id = None
76
+
77
+ def forward(
78
+ self,
79
+ pixel_values: torch.FloatTensor,
80
+ input_ids: torch.LongTensor = None,
81
+ attention_mask: Optional[torch.Tensor] = None,
82
+ position_ids: Optional[torch.LongTensor] = None,
83
+ image_flags: Optional[torch.LongTensor] = None,
84
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
85
+ labels: Optional[torch.LongTensor] = None,
86
+ use_cache: Optional[bool] = None,
87
+ output_attentions: Optional[bool] = None,
88
+ output_hidden_states: Optional[bool] = None,
89
+ return_dict: Optional[bool] = None,
90
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
91
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
92
+
93
+ image_flags = image_flags.squeeze(-1)
94
+ input_embeds = self.language_model.get_input_embeddings()(input_ids)
95
+
96
+ vit_embeds = self.extract_feature(pixel_values)
97
+ vit_embeds = vit_embeds[image_flags == 1]
98
+ vit_batch_size = pixel_values.shape[0]
99
+
100
+ B, N, C = input_embeds.shape
101
+ input_embeds = input_embeds.reshape(B * N, C)
102
+
103
+ if torch.distributed.get_rank() == 0:
104
+ print(f'dynamic ViT batch size: {vit_batch_size}, images per sample: {vit_batch_size / B}, dynamic token length: {N}')
105
+
106
+ input_ids = input_ids.reshape(B * N)
107
+ selected = (input_ids == self.img_context_token_id)
108
+ try:
109
+ input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds.reshape(-1, C)
110
+ except Exception as e:
111
+ vit_embeds = vit_embeds.reshape(-1, C)
112
+ print(f'warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, '
113
+ f'vit_embeds.shape={vit_embeds.shape}')
114
+ n_token = selected.sum()
115
+ input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds[:n_token]
116
+
117
+ input_embeds = input_embeds.reshape(B, N, C)
118
+
119
+ outputs = self.language_model(
120
+ inputs_embeds=input_embeds,
121
+ attention_mask=attention_mask,
122
+ position_ids=position_ids,
123
+ past_key_values=past_key_values,
124
+ use_cache=use_cache,
125
+ output_attentions=output_attentions,
126
+ output_hidden_states=output_hidden_states,
127
+ return_dict=return_dict,
128
+ )
129
+ logits = outputs.logits
130
+
131
+ loss = None
132
+ if labels is not None:
133
+ # Shift so that tokens < n predict n
134
+ shift_logits = logits[..., :-1, :].contiguous()
135
+ shift_labels = labels[..., 1:].contiguous()
136
+ # Flatten the tokens
137
+ loss_fct = CrossEntropyLoss()
138
+ shift_logits = shift_logits.view(-1, self.language_model.config.vocab_size)
139
+ shift_labels = shift_labels.view(-1)
140
+ # Enable model parallelism
141
+ shift_labels = shift_labels.to(shift_logits.device)
142
+ loss = loss_fct(shift_logits, shift_labels)
143
+
144
+ if not return_dict:
145
+ output = (logits,) + outputs[1:]
146
+ return (loss,) + output if loss is not None else output
147
+
148
+ return CausalLMOutputWithPast(
149
+ loss=loss,
150
+ logits=logits,
151
+ past_key_values=outputs.past_key_values,
152
+ hidden_states=outputs.hidden_states,
153
+ attentions=outputs.attentions,
154
+ )
155
+
156
+ def pixel_shuffle(self, x, scale_factor=0.5):
157
+ n, w, h, c = x.size()
158
+ # N, W, H, C --> N, W, H * scale, C // scale
159
+ x = x.view(n, w, int(h * scale_factor), int(c / scale_factor))
160
+ # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
161
+ x = x.permute(0, 2, 1, 3).contiguous()
162
+ # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
163
+ x = x.view(n, int(h * scale_factor), int(w * scale_factor),
164
+ int(c / (scale_factor * scale_factor)))
165
+ if self.ps_version == 'v1':
166
+ warnings.warn("In ps_version 'v1', the height and width have not been swapped back, "
167
+ 'which results in a transposed image.')
168
+ else:
169
+ x = x.permute(0, 2, 1, 3).contiguous()
170
+ return x
171
+
172
+ def extract_feature(self, pixel_values):
173
+ if self.select_layer == -1:
174
+ vit_embeds = self.vision_model(
175
+ pixel_values=pixel_values,
176
+ output_hidden_states=False,
177
+ return_dict=True).last_hidden_state
178
+ else:
179
+ vit_embeds = self.vision_model(
180
+ pixel_values=pixel_values,
181
+ output_hidden_states=True,
182
+ return_dict=True).hidden_states[self.select_layer]
183
+ vit_embeds = vit_embeds[:, 1:, :]
184
+
185
+ h = w = int(vit_embeds.shape[1] ** 0.5)
186
+ vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], h, w, -1)
187
+ vit_embeds = self.pixel_shuffle(vit_embeds, scale_factor=self.downsample_ratio)
188
+ vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], -1, vit_embeds.shape[-1])
189
+ vit_embeds = self.mlp1(vit_embeds)
190
+ return vit_embeds
191
+
192
+ def batch_chat(self, tokenizer, pixel_values, questions, generation_config, num_patches_list=None,
193
+ history=None, return_history=False, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>',
194
+ IMG_CONTEXT_TOKEN='<IMG_CONTEXT>', verbose=False, image_counts=None):
195
+ if history is not None or return_history:
196
+ print('Now multi-turn chat is not supported in batch_chat.')
197
+ raise NotImplementedError
198
+
199
+ if image_counts is not None:
200
+ num_patches_list = image_counts
201
+ print('Warning: `image_counts` is deprecated. Please use `num_patches_list` instead.')
202
+
203
+ img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
204
+ self.img_context_token_id = img_context_token_id
205
+
206
+ if verbose and pixel_values is not None:
207
+ image_bs = pixel_values.shape[0]
208
+ print(f'dynamic ViT batch size: {image_bs}')
209
+
210
+ queries = []
211
+ for idx, num_patches in enumerate(num_patches_list):
212
+ question = questions[idx]
213
+ if pixel_values is not None and '<image>' not in question:
214
+ question = '<image>\n' + question
215
+ template = get_conv_template(self.template)
216
+ template.append_message(template.roles[0], question)
217
+ template.append_message(template.roles[1], None)
218
+ query = template.get_prompt()
219
+
220
+ image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
221
+ query = query.replace('<image>', image_tokens, 1)
222
+ queries.append(query)
223
+
224
+ tokenizer.padding_side = 'left'
225
+ model_inputs = tokenizer(queries, return_tensors='pt', padding=True)
226
+ input_ids = model_inputs['input_ids'].cuda()
227
+ attention_mask = model_inputs['attention_mask'].cuda()
228
+ eos_token_id = tokenizer.convert_tokens_to_ids(template.sep)
229
+ generation_config['eos_token_id'] = eos_token_id
230
+ generation_output = self.generate(
231
+ pixel_values=pixel_values,
232
+ input_ids=input_ids,
233
+ attention_mask=attention_mask,
234
+ **generation_config
235
+ )
236
+ responses = tokenizer.batch_decode(generation_output, skip_special_tokens=True)
237
+ responses = [response.split(template.sep)[0].strip() for response in responses]
238
+ return responses
239
+
240
+ def chat(self, tokenizer, pixel_values, question, generation_config, history=None, return_history=False,
241
+ num_patches_list=None, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>', IMG_CONTEXT_TOKEN='<IMG_CONTEXT>',
242
+ verbose=False):
243
+
244
+ if history is None and pixel_values is not None and '<image>' not in question:
245
+ question = '<image>\n' + question
246
+
247
+ if num_patches_list is None:
248
+ num_patches_list = [pixel_values.shape[0]] if pixel_values is not None else []
249
+ assert pixel_values is None or len(pixel_values) == sum(num_patches_list)
250
+
251
+ img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
252
+ self.img_context_token_id = img_context_token_id
253
+
254
+ template = get_conv_template(self.template)
255
+ eos_token_id = tokenizer.convert_tokens_to_ids(template.sep)
256
+
257
+ history = [] if history is None else history
258
+ for (old_question, old_answer) in history:
259
+ template.append_message(template.roles[0], old_question)
260
+ template.append_message(template.roles[1], old_answer)
261
+ template.append_message(template.roles[0], question)
262
+ template.append_message(template.roles[1], None)
263
+ query = template.get_prompt()
264
+
265
+ if verbose and pixel_values is not None:
266
+ image_bs = pixel_values.shape[0]
267
+ print(f'dynamic ViT batch size: {image_bs}')
268
+
269
+ for num_patches in num_patches_list:
270
+ image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
271
+ query = query.replace('<image>', image_tokens, 1)
272
+
273
+ model_inputs = tokenizer(query, return_tensors='pt')
274
+ input_ids = model_inputs['input_ids'].cuda()
275
+ attention_mask = model_inputs['attention_mask'].cuda()
276
+ generation_config['eos_token_id'] = eos_token_id
277
+ generation_output = self.generate(
278
+ pixel_values=pixel_values,
279
+ input_ids=input_ids,
280
+ attention_mask=attention_mask,
281
+ **generation_config
282
+ )
283
+ response = tokenizer.batch_decode(generation_output, skip_special_tokens=True)[0]
284
+ response = response.split(template.sep)[0].strip()
285
+ history.append((question, response))
286
+ if return_history:
287
+ return response, history
288
+ else:
289
+ query_to_print = query.replace(IMG_CONTEXT_TOKEN, '')
290
+ query_to_print = query_to_print.replace(f'{IMG_START_TOKEN}{IMG_END_TOKEN}', '<image>')
291
+ if verbose:
292
+ print(query_to_print, response)
293
+ return response
294
+
295
+ @torch.no_grad()
296
+ def generate(
297
+ self,
298
+ pixel_values: Optional[torch.FloatTensor] = None,
299
+ input_ids: Optional[torch.FloatTensor] = None,
300
+ attention_mask: Optional[torch.LongTensor] = None,
301
+ visual_features: Optional[torch.FloatTensor] = None,
302
+ generation_config: Optional[GenerationConfig] = None,
303
+ output_hidden_states: Optional[bool] = None,
304
+ return_dict: Optional[bool] = None,
305
+ **generate_kwargs,
306
+ ) -> torch.LongTensor:
307
+
308
+ assert self.img_context_token_id is not None
309
+ if pixel_values is not None:
310
+ if visual_features is not None:
311
+ vit_embeds = visual_features
312
+ else:
313
+ vit_embeds = self.extract_feature(pixel_values)
314
+ input_embeds = self.language_model.get_input_embeddings()(input_ids)
315
+ B, N, C = input_embeds.shape
316
+ input_embeds = input_embeds.reshape(B * N, C)
317
+
318
+ input_ids = input_ids.reshape(B * N)
319
+ selected = (input_ids == self.img_context_token_id)
320
+ assert selected.sum() != 0
321
+ input_embeds[selected] = vit_embeds.reshape(-1, C).to(input_embeds.device)
322
+
323
+ input_embeds = input_embeds.reshape(B, N, C)
324
+ else:
325
+ input_embeds = self.language_model.get_input_embeddings()(input_ids)
326
+
327
+ outputs = self.language_model.generate(
328
+ inputs_embeds=input_embeds,
329
+ attention_mask=attention_mask,
330
+ generation_config=generation_config,
331
+ output_hidden_states=output_hidden_states,
332
+ return_dict=return_dict,
333
+ use_cache=True,
334
+ **generate_kwargs,
335
+ )
336
+
337
+ return outputs
special_tokens_map.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<img>",
4
+ "</img>",
5
+ "<IMG_CONTEXT>",
6
+ "<quad>",
7
+ "</quad>",
8
+ "<ref>",
9
+ "</ref>",
10
+ "<box>",
11
+ "</box>"
12
+ ],
13
+ "bos_token": {
14
+ "content": "<|startoftext|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false
19
+ },
20
+ "eos_token": {
21
+ "content": "<|im_end|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false
26
+ },
27
+ "pad_token": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false
33
+ },
34
+ "unk_token": {
35
+ "content": "<unk>",
36
+ "lstrip": false,
37
+ "normalized": false,
38
+ "rstrip": false,
39
+ "single_word": false
40
+ }
41
+ }
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:386c49cf943d71aa110361135338c50e38beeff0a66593480421f37b319e1a39
3
+ size 1033105
tokenizer_config.json ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "0": {
6
+ "content": "<unk>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "1": {
14
+ "content": "<|startoftext|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "2": {
22
+ "content": "<|endoftext|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ },
29
+ "6": {
30
+ "content": "<|im_start|>",
31
+ "lstrip": false,
32
+ "normalized": false,
33
+ "rstrip": false,
34
+ "single_word": false,
35
+ "special": false
36
+ },
37
+ "7": {
38
+ "content": "<|im_end|>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false,
43
+ "special": true
44
+ },
45
+ "68": {
46
+ "content": "<img>",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false,
51
+ "special": true
52
+ },
53
+ "70": {
54
+ "content": "</img>",
55
+ "lstrip": false,
56
+ "normalized": false,
57
+ "rstrip": false,
58
+ "single_word": false,
59
+ "special": true
60
+ },
61
+ "64000": {
62
+ "content": "<IMG_CONTEXT>",
63
+ "lstrip": false,
64
+ "normalized": false,
65
+ "rstrip": false,
66
+ "single_word": false,
67
+ "special": true
68
+ },
69
+ "64001": {
70
+ "content": "<quad>",
71
+ "lstrip": false,
72
+ "normalized": false,
73
+ "rstrip": false,
74
+ "single_word": false,
75
+ "special": true
76
+ },
77
+ "64002": {
78
+ "content": "</quad>",
79
+ "lstrip": false,
80
+ "normalized": false,
81
+ "rstrip": false,
82
+ "single_word": false,
83
+ "special": true
84
+ },
85
+ "64003": {
86
+ "content": "<ref>",
87
+ "lstrip": false,
88
+ "normalized": false,
89
+ "rstrip": false,
90
+ "single_word": false,
91
+ "special": true
92
+ },
93
+ "64004": {
94
+ "content": "</ref>",
95
+ "lstrip": false,
96
+ "normalized": false,
97
+ "rstrip": false,
98
+ "single_word": false,
99
+ "special": true
100
+ },
101
+ "64005": {
102
+ "content": "<box>",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false,
107
+ "special": true
108
+ },
109
+ "64006": {
110
+ "content": "</box>",
111
+ "lstrip": false,
112
+ "normalized": false,
113
+ "rstrip": false,
114
+ "single_word": false,
115
+ "special": true
116
+ }
117
+ },
118
+ "additional_special_tokens": [
119
+ "<img>",
120
+ "</img>",
121
+ "<IMG_CONTEXT>",
122
+ "<quad>",
123
+ "</quad>",
124
+ "<ref>",
125
+ "</ref>",
126
+ "<box>",
127
+ "</box>"
128
+ ],
129
+ "bos_token": "<|startoftext|>",
130
+ "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
131
+ "clean_up_tokenization_spaces": false,
132
+ "eos_token": "<|im_end|>",
133
+ "legacy": true,
134
+ "model_max_length": 8192,
135
+ "pad_token": "<unk>",
136
+ "sp_model_kwargs": {},
137
+ "spaces_between_special_tokens": false,
138
+ "tokenizer_class": "LlamaTokenizer",
139
+ "trust_remote_code": false,
140
+ "unk_token": "<unk>",
141
+ "use_default_system_prompt": false,
142
+ "use_fast": true
143
+ }