prithivMLmods commited on
Commit
bd5078a
1 Parent(s): 17431e4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +130 -1
README.md CHANGED
@@ -11,4 +11,133 @@ tags:
11
  - Language
12
  - VLM
13
  - Character-Anology
14
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  - Language
12
  - VLM
13
  - Character-Anology
14
+ ---
15
+ # Qwen2-VL-Ocrtest-2B-Instruct
16
+
17
+ ![11.png](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/zBC0BSRyGqcujikXrWqC6.png)
18
+
19
+ The **Qwen2-VL-Ocrtest-2B-Instruct** model is a fine-tuned version of **Qwen/Qwen2-VL-2B-Instruct**, tailored for tasks that involve **Optical Character Recognition (OCR)**, **image-to-text conversion**, and **math problem solving with LaTeX formatting**. This model integrates a conversational approach with visual and textual understanding to handle multi-modal tasks effectively.
20
+
21
+ #### Key Enhancements:
22
+
23
+ * **SoTA understanding of images of various resolution & ratio**: Qwen2-VL achieves state-of-the-art performance on visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, MTVQA, etc.
24
+
25
+ * **Understanding videos of 20min+**: Qwen2-VL can understand videos over 20 minutes for high-quality video-based question answering, dialog, content creation, etc.
26
+
27
+ * **Agent that can operate your mobiles, robots, etc.**: with the abilities of complex reasoning and decision making, Qwen2-VL can be integrated with devices like mobile phones, robots, etc., for automatic operation based on visual environment and text instructions.
28
+
29
+ * **Multilingual Support**: to serve global users, besides English and Chinese, Qwen2-VL now supports the understanding of texts in different languages inside images, including most European languages, Japanese, Korean, Arabic, Vietnamese, etc.
30
+
31
+ | **File Name** | **Size** | **Description** | **Upload Status** |
32
+ |---------------------------|------------|------------------------------------------------|-------------------|
33
+ | `.gitattributes` | 1.52 kB | Configures LFS tracking for specific model files. | Initial commit |
34
+ | `README.md` | 203 Bytes | Minimal details about the uploaded model. | Updated |
35
+ | `added_tokens.json` | 408 Bytes | Additional tokens used by the model tokenizer. | Uploaded |
36
+ | `chat_template.json` | 1.05 kB | Template for chat-based model input/output. | Uploaded |
37
+ | `config.json` | 1.24 kB | Model configuration metadata. | Uploaded |
38
+ | `generation_config.json` | 252 Bytes | Configuration for text generation settings. | Uploaded |
39
+ | `merges.txt` | 1.82 MB | BPE merge rules for tokenization. | Uploaded |
40
+ | `model.safetensors` | 4.42 GB | Serialized model weights in a secure format. | Uploaded (LFS) |
41
+ | `preprocessor_config.json`| 596 Bytes | Preprocessing configuration for input data. | Uploaded |
42
+ | `vocab.json` | 2.78 MB | Vocabulary file for tokenization. | Uploaded |
43
+
44
+ ---
45
+ ### How to Use
46
+
47
+ ```python
48
+ from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
49
+ from qwen_vl_utils import process_vision_info
50
+
51
+ # default: Load the model on the available device(s)
52
+ model = Qwen2VLForConditionalGeneration.from_pretrained(
53
+ "prithivMLmods/Qwen2-VL-Ocrtest-2B-Instruct", torch_dtype="auto", device_map="auto"
54
+ )
55
+
56
+ # We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
57
+ # model = Qwen2VLForConditionalGeneration.from_pretrained(
58
+ # "prithivMLmods/Qwen2-VL-Ocrtest-2B-Instruct",
59
+ # torch_dtype=torch.bfloat16,
60
+ # attn_implementation="flash_attention_2",
61
+ # device_map="auto",
62
+ # )
63
+
64
+ # default processer
65
+ processor = AutoProcessor.from_pretrained("prithivMLmods/Qwen2-VL-Ocrtest-2B-Instruct")
66
+
67
+ # The default range for the number of visual tokens per image in the model is 4-16384. You can set min_pixels and max_pixels according to your needs, such as a token count range of 256-1280, to balance speed and memory usage.
68
+ # min_pixels = 256*28*28
69
+ # max_pixels = 1280*28*28
70
+ # processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)
71
+
72
+ messages = [
73
+ {
74
+ "role": "user",
75
+ "content": [
76
+ {
77
+ "type": "image",
78
+ "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
79
+ },
80
+ {"type": "text", "text": "Describe this image."},
81
+ ],
82
+ }
83
+ ]
84
+
85
+ # Preparation for inference
86
+ text = processor.apply_chat_template(
87
+ messages, tokenize=False, add_generation_prompt=True
88
+ )
89
+ image_inputs, video_inputs = process_vision_info(messages)
90
+ inputs = processor(
91
+ text=[text],
92
+ images=image_inputs,
93
+ videos=video_inputs,
94
+ padding=True,
95
+ return_tensors="pt",
96
+ )
97
+ inputs = inputs.to("cuda")
98
+
99
+ # Inference: Generation of the output
100
+ generated_ids = model.generate(**inputs, max_new_tokens=128)
101
+ generated_ids_trimmed = [
102
+ out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
103
+ ]
104
+ output_text = processor.batch_decode(
105
+ generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
106
+ )
107
+ print(output_text)
108
+ ```
109
+
110
+ ### **Key Features**
111
+
112
+ 1. **Vision-Language Integration:**
113
+ - Combines **image understanding** with **natural language processing** to convert images into text.
114
+
115
+ 2. **Optical Character Recognition (OCR):**
116
+ - Extracts and processes textual information from images with high accuracy.
117
+
118
+ 3. **Math and LaTeX Support:**
119
+ - Solves math problems and outputs equations in **LaTeX format**.
120
+
121
+ 4. **Conversational Capabilities:**
122
+ - Designed to handle **multi-turn interactions**, providing context-aware responses.
123
+
124
+ 5. **Image-Text-to-Text Generation:**
125
+ - Inputs can include **images, text, or a combination**, and the model generates descriptive or problem-solving text.
126
+
127
+ 6. **Secure Weight Format:**
128
+ - Uses **Safetensors** for faster and more secure model weight loading.
129
+
130
+ ---
131
+
132
+ ### **Training Details**
133
+
134
+ - **Base Model:** [Qwen/Qwen2-VL-2B-Instruct](#)
135
+ - **Model Size:**
136
+ - 2.21 Billion parameters
137
+ - Optimized for **BF16** tensor type, enabling efficient inference.
138
+
139
+ - **Specializations:**
140
+ - OCR tasks in images containing text.
141
+ - Mathematical reasoning and LaTeX output for equations.
142
+
143
+ ---