Hugging Face's logo

OpenGVLab
/
InternVL2-1B

Image-Text-to-Text
Transformers
Safetensors
multilingual
internvl_chat
feature-extraction
internvl
vision
ocr
multi-image
video
custom_code
conversational
Model card Files Community
7
czczup commited on
Commit
0706313
•
1 Parent(s): d3da944

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +66 -42
README.md CHANGED
@@ -9,6 +9,8 @@ pipeline_tag: image-text-to-text
9
 
10
  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971) \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
11
 
 
 
12
  ## Introduction
13
 
14
  We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-1B model.
@@ -17,6 +19,16 @@ Compared to the state-of-the-art open-source multimodal large language models, I
17
 
18
  InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
19
 
 
 
 
 
 
 
 
 
 
 
20
  ## Model Details
21
 
22
  InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-1B consists of [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px), an MLP projector, and [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct).
@@ -25,27 +37,28 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
25
 
26
  ### Image Benchmarks
27
 
28
- | Benchmark | PaliGemma-3B | Mini-InternVL-2B-1.5 | InternVL2-2B | InternVL2-1B |
29
- | :--------------------------: | :----------: | :------------------: | :----------: | :----------: |
30
- | Model Size | 2.9B | 2.2B | 2.2B | 0.9B |
31
- | | | | | |
32
- | DocVQA<sub>test</sub> | - | 85.0 | 86.9 | 81.7 |
33
- | ChartQA<sub>test</sub> | - | 74.8 | 76.2 | 72.9 |
34
- | InfoVQA<sub>test</sub> | - | 55.4 | 58.9 | 50.9 |
35
- | TextVQA<sub>val</sub> | 68.1 | 70.5 | 73.4 | 70.5 |
36
- | OCRBench | 614 | 654 | 784 | 754 |
37
- | MME<sub>sum</sub> | 1686.1 | 1901.5 | 1876.8 | 1794.4 |
38
- | RealWorldQA | 55.2 | 57.9 | 57.3 | 50.3 |
39
- | AI2D<sub>test</sub> | 68.3 | 69.8 | 74.1 | 64.1 |
40
- | MMMU<sub>val</sub> | 34.9 | 34.6 / 37.4 | 34.3 / 36.3 | 35.4 / 36.7 |
41
- | MMBench-EN<sub>test</sub> | 71.0 | 70.9 | 73.2 | 65.4 |
42
- | MMBench-CN<sub>test</sub> | 63.6 | 66.2 | 70.9 | 60.7 |
43
- | CCBench<sub>dev</sub> | 29.6 | 63.5 | 74.7 | 75.7 |
44
- | MMVet<sub>GPT-4-0613</sub> | - | 39.3 | 44.6 | 37.8 |
45
- | MMVet<sub>GPT-4-Turbo</sub> | 33.1 | 35.5 | 39.5 | 37.3 |
46
- | SEED-Image | 69.6 | 69.8 | 71.6 | 65.6 |
47
- | HallBench<sub>avg</sub> | 32.2 | 37.5 | 37.9 | 33.4 |
48
- | MathVista<sub>testmini</sub> | 28.7 | 41.1 | 46.3 | 37.7 |
 
49
 
50
  - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
51
 
@@ -393,6 +406,16 @@ If you find this project useful in your research, please consider citing:
393
 
394
  InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
395
 
 
 
 
 
 
 
 
 
 
 
396
  ## 模型细节
397
 
398
  InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-1B 包含 [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)、一个 MLP 投影器和 [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)。
@@ -401,27 +424,28 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
401
 
402
  ### 图像相关评测
403
 
404
- | 评测数据集 | PaliGemma-3B | Mini-InternVL-2B-1.5 | InternVL2-2B | InternVL2-1B |
405
- | :--------------------------: | :----------: | :------------------: | :----------: | :----------: |
406
- | 模型大小 | 2.9B | 2.2B | 2.2B | 0.9B |
407
- | | | | | |
408
- | DocVQA<sub>test</sub> | - | 85.0 | 86.9 | 81.7 |
409
- | ChartQA<sub>test</sub> | - | 74.8 | 76.2 | 72.9 |
410
- | InfoVQA<sub>test</sub> | - | 55.4 | 58.9 | 50.9 |
411
- | TextVQA<sub>val</sub> | 68.1 | 70.5 | 73.4 | 70.5 |
412
- | OCRBench | 614 | 654 | 784 | 754 |
413
- | MME<sub>sum</sub> | 1686.1 | 1901.5 | 1876.8 | 1794.4 |
414
- | RealWorldQA | 55.2 | 57.9 | 57.3 | 50.3 |
415
- | AI2D<sub>test</sub> | 68.3 | 69.8 | 74.1 | 64.1 |
416
- | MMMU<sub>val</sub> | 34.9 | 34.6 / 37.4 | 34.3 / 36.3 | 35.4 / 36.7 |
417
- | MMBench-EN<sub>test</sub> | 71.0 | 70.9 | 73.2 | 65.4 |
418
- | MMBench-CN<sub>test</sub> | 63.6 | 66.2 | 70.9 | 60.7 |
419
- | CCBench<sub>dev</sub> | 29.6 | 63.5 | 74.7 | 75.7 |
420
- | MMVet<sub>GPT-4-0613</sub> | - | 39.3 | 44.6 | 37.8 |
421
- | MMVet<sub>GPT-4-Turbo</sub> | 33.1 | 35.5 | 39.5 | 37.3 |
422
- | SEED-Image | 69.6 | 69.8 | 71.6 | 65.6 |
423
- | HallBench<sub>avg</sub> | 32.2 | 37.5 | 37.9 | 33.4 |
424
- | MathVista<sub>testmini</sub> | 28.7 | 41.1 | 46.3 | 37.7 |
 
425
 
426
  - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
427
 
 
9
 
10
  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971) \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
11
 
12
+ [切换至中文版](#简介)
13
+
14
  ## Introduction
15
 
16
  We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-1B model.
 
19
 
20
  InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
21
 
22
+ | Model Name | Vision Part | Language Part | HF Link | MS Link |
23
+ | :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
24
+ | InternVL2-1B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B) |
25
+ | InternVL2-2B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B) |
26
+ | InternVL2-4B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B) |
27
+ | InternVL2-8B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B) |
28
+ | InternVL2-26B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B) |
29
+ | InternVL2-40B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B) |
30
+ | InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
31
+
32
  ## Model Details
33
 
34
  InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-1B consists of [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px), an MLP projector, and [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct).
 
37
 
38
  ### Image Benchmarks
39
 
40
+ | Benchmark | PaliGemma-3B | Mini-InternVL-2B-1-5 | InternVL2-2B | InternVL2-1B |
41
+ | :-----------------------------: | :----------: | :------------------: | :----------: | :----------: |
42
+ | Model Size | 2.9B | 2.2B | 2.2B | 0.9B |
43
+ | | | | | |
44
+ | DocVQA<sub>test</sub> | - | 85.0 | 86.9 | 81.7 |
45
+ | ChartQA<sub>test</sub> | - | 74.8 | 76.2 | 72.9 |
46
+ | InfoVQA<sub>test</sub> | - | 55.4 | 58.9 | 50.9 |
47
+ | TextVQA<sub>val</sub> | 68.1 | 70.5 | 73.4 | 70.5 |
48
+ | OCRBench | 614 | 654 | 784 | 754 |
49
+ | MME<sub>sum</sub> | 1686.1 | 1901.5 | 1876.8 | 1794.4 |
50
+ | RealWorldQA | 55.2 | 57.9 | 57.3 | 50.3 |
51
+ | AI2D<sub>test</sub> | 68.3 | 69.8 | 74.1 | 64.1 |
52
+ | MMMU<sub>val</sub> | 34.9 | 34.6 / 37.4 | 34.3 / 36.3 | 35.4 / 36.7 |
53
+ | MMBench-EN<sub>test</sub> | 71.0 | 70.9 | 73.2 | 65.4 |
54
+ | MMBench-CN<sub>test</sub> | 63.6 | 66.2 | 70.9 | 60.7 |
55
+ | CCBench<sub>dev</sub> | 29.6 | 63.5 | 74.7 | 75.7 |
56
+ | MMVet<sub>GPT-4-0613</sub> | - | 39.3 | 44.6 | 37.8 |
57
+ | MMVet<sub>GPT-4-Turbo</sub> | 33.1 | 35.5 | 39.5 | 33.3 |
58
+ | SEED-Image | 69.6 | 69.8 | 71.6 | 65.6 |
59
+ | HallBench<sub>avg</sub> | 32.2 | 37.5 | 37.9 | 33.4 |
60
+ | MathVista<sub>testmini</sub> | 28.7 | 41.1 | 46.3 | 37.7 |
61
+ | OpenCompass<sub>avg-score</sub> | 46.6 | 49.8 | 54.0 | 48.3 |
62
 
63
  - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
64
 
 
406
 
407
  InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
408
 
409
+ | 模型名称 | 视觉部分 | 语言部分 | HF 链��� | MS 链接 |
410
+ | :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
411
+ | InternVL2-1B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B) |
412
+ | InternVL2-2B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B) |
413
+ | InternVL2-4B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B) |
414
+ | InternVL2-8B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B) |
415
+ | InternVL2-26B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B) |
416
+ | InternVL2-40B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B) |
417
+ | InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
418
+
419
  ## 模型细节
420
 
421
  InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-1B 包含 [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)、一个 MLP 投影器和 [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)。
 
424
 
425
  ### 图像相关评测
426
 
427
+ | 评测数据集 | PaliGemma-3B | Mini-InternVL-2B-1-5 | InternVL2-2B | InternVL2-1B |
428
+ | :-----------------------------: | :----------: | :------------------: | :----------: | :----------: |
429
+ | 模型大小 | 2.9B | 2.2B | 2.2B | 0.9B |
430
+ | | | | | |
431
+ | DocVQA<sub>test</sub> | - | 85.0 | 86.9 | 81.7 |
432
+ | ChartQA<sub>test</sub> | - | 74.8 | 76.2 | 72.9 |
433
+ | InfoVQA<sub>test</sub> | - | 55.4 | 58.9 | 50.9 |
434
+ | TextVQA<sub>val</sub> | 68.1 | 70.5 | 73.4 | 70.5 |
435
+ | OCRBench | 614 | 654 | 784 | 754 |
436
+ | MME<sub>sum</sub> | 1686.1 | 1901.5 | 1876.8 | 1794.4 |
437
+ | RealWorldQA | 55.2 | 57.9 | 57.3 | 50.3 |
438
+ | AI2D<sub>test</sub> | 68.3 | 69.8 | 74.1 | 64.1 |
439
+ | MMMU<sub>val</sub> | 34.9 | 34.6 / 37.4 | 34.3 / 36.3 | 35.4 / 36.7 |
440
+ | MMBench-EN<sub>test</sub> | 71.0 | 70.9 | 73.2 | 65.4 |
441
+ | MMBench-CN<sub>test</sub> | 63.6 | 66.2 | 70.9 | 60.7 |
442
+ | CCBench<sub>dev</sub> | 29.6 | 63.5 | 74.7 | 75.7 |
443
+ | MMVet<sub>GPT-4-0613</sub> | - | 39.3 | 44.6 | 37.8 |
444
+ | MMVet<sub>GPT-4-Turbo</sub> | 33.1 | 35.5 | 39.5 | 37.3 |
445
+ | SEED-Image | 69.6 | 69.8 | 71.6 | 65.6 |
446
+ | HallBench<sub>avg</sub> | 32.2 | 37.5 | 37.9 | 33.4 |
447
+ | MathVista<sub>testmini</sub> | 28.7 | 41.1 | 46.3 | 37.7 |
448
+ | OpenCompass<sub>avg-score</sub> | 46.6 | 49.8 | 54.0 | 48.3 |
449
 
450
  - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
451