Update README.md

README.md

@@ -7,31 +7,42 @@ datasets:
 
 
 ## OmniLMM 12B
-**OmniLMM-12B** is the most capable version. The model is built based on [EVA02-5B](https://github.com/baaivision/EVA/tree/master/EVA-CLIP) and [Zephyr-7B-β](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), connected with a perceiver resampler layer, and trained on multimodal data in a curriculum fashion. The model has three notable features:
+
+**OmniLMM-12B** is the most capable version of OmniLMM currently. The model is built based on EVA02-5B and Zephyr-7B-β, connected with a perceiver resampler layer, and trained on multimodal data in a curriculum fashion. The model has three notable features:
 
 - 🔥 **Strong Performance.** 
 
-  OmniLMM-12B achieves **leading performance** among models with comparable sizes, surpassing established LMMs on multiple benchmarks (including MME, MMBench, SEED-Bench, etc). The model also **supports OCR capability** and endows **rich multimodal world knowledge**.
+  OmniLMM-12B achieves **leading performance** among models with comparable sizes, surpassing established LMMs on multiple benchmarks (including MME, MMBench, SEED-Bench, etc). The model also endows rich multi-modal world knowledge.
 
 - 🏆 **Trustworthy Behavior.** 
 
-  LMMs are known for suffering from hallucination, often generating text that is not factually grounded in images (e.g., faithfully describing non-existing objects in images). OmniLMM-12B is **the first state-of-the-art open-source LMM aligned via multimodal RLHF for trustworthy behavior** (using our recent [RLHF-V](https://rlhf-v.github.io/) technique) and **ranked #1** among open-source models on [MMHal-Bench](https://huggingface.co/datasets/Shengcao1006/MMHal-Bench).
+  LMMs are known for suffering from hallucination, often generating text that is not factually grounded in images (e.g., faithfully describing non-existing objects in images). OmniLMM-12B is **the first state-of-the-art open-source LMM aligned via multimodal RLHF for trustworthy behavior** (using the recent [RLHF-V](https://rlhf-v.github.io/) technique). It **ranks #1** among open-source models on [MMHal-Bench](https://huggingface.co/datasets/Shengcao1006/MMHal-Bench), and **outperforms GPT-4V** on [Object HalBench](https://arxiv.org/abs/2312.00849).
 
 - 🕹 **Real-time Multimodal Interaction.** 
 
-  We combine the OmniLMM-12B and GPT-3.5 into a **real-time multimodal interactive assistant**. The assistant accepts video streams from the camera and speech streams from the microphone and emits speech output. While still primary, we find the model can **replicate some of the fun cases shown in the Gemini Demo video, without any video edition**.
+  We combine the OmniLMM-12B and GPT-3.5 (text-only) into a **real-time multimodal interactive assistant**. The assistant accepts video streams from the camera and speech streams from the microphone and emits speech output. While still primary, we find the model can **replicate some of the fun cases shown in the Gemini Demo video, without any video edition**.
+
 
 ## Evaluation
+
+
+<div align="center">
+    <img src=https://cdn-uploads.huggingface.co/production/uploads/64abc4aa6cadc7aca585dddf/LuKikSY4CJiqtHocGP_xu.png width=66% />
+</div>
+<details>
+<summary>Click to view results on MME, MMBench, MMMU, MMBench, MMHal-Bench, Object HalBench, SeedBench, LLaVA Bench W, MathVista. </summary>
+
 <table>
 <thead>
   <tr>
     <th align="left">Model</th>
     <th>Size</th>
     <th>MME</th>
+    <th nowrap="nowrap">MMB dev (en)</th>
     <th nowrap="nowrap" >MMMU val</th>
     <th nowrap="nowrap" >MMHal-Bench</th>
+    <th nowrap="nowrap" >Object HalBench</th>
     <th nowrap="nowrap" >SeedBench-I</th>
-    <th nowrap="nowrap">MMB dev (en)</th>
     <th>MathVista</th>
     <th nowrap="nowrap" >LLaVA Bench W</th>
   </tr>
@@ -41,10 +52,11 @@ datasets:
     <td align="left">GPT-4V†</td>
     <td>-</td>
     <td>1409</td>
+    <td>75.1 </td>
     <td>56.8</td>
     <td>3.53 / 70.8</td>
+    <td>86.4 / 92.7</td>
     <td>71.6 </td>
-    <td>75.1 </td>
     <td>47.8 </td>
     <td>93.1 </td>
   </tr>
@@ -52,10 +64,11 @@ datasets:
     <td nowrap="nowrap" align="left">Qwen-VL-Plus†</td>
     <td>-</td>
     <td>1681</td>
+    <td>66.2 </td>
     <td>45.2</td>
     <td>- </td>
+    <td>- </td>
     <td>65.7 </td>
-    <td>66.2 </td>
     <td>36.0 </td>
     <td>73.7 </td>
   </tr>
@@ -63,10 +76,11 @@ datasets:
     <td align="left">Yi-VL 6B</td>
     <td align="right">6.7B </td>
     <td>- </td>
+    <td>68.2 </td>
     <td>39.1 </td>
     <td>- </td>
+    <td>- </td>
     <td>66.1 </td>
-    <td>68.2 </td>
     <td>28.0 </td>
     <td>39.9 </td>
   </tr>
@@ -74,10 +88,11 @@ datasets:
     <td nowrap="nowrap" align="left" >Qwen-VL-Chat</td>
     <td align="right">9.6B</td>
     <td>1488</td>
+    <td>60.6 </td>
     <td>35.9</td>
     <td>2.93 / 59.4</td>
+    <td>56.2 / 80.0</td>
     <td>64.8 </td>
-    <td>60.6 </td>
     <td>33.8 </td>
     <td>67.7 </td>
   </tr>
@@ -85,10 +100,11 @@ datasets:
     <td align="left" >CogVLM</td>
     <td align="right">17.4B</td>
     <td>1438</td>
+    <td>63.7 </td>
     <td>32.1 </td>
     <td>2.68 / 52.1 </td>
+    <td>73.6 / 87.4 </td>
     <td>68.8 </td>
-    <td>63.7 </td>
     <td>34.7 </td>
     <td>73.9 </td>
   </tr>
@@ -96,10 +112,11 @@ datasets:
     <td align="left" >LLaVA 1.5</td>
     <td align="right">13.6B </td>
     <td>1531 </td>
+    <td>68.2 </td>
     <td>36.4 </td>
     <td>2.71 / 51.0 </td>
+    <td>53.7 / 77.4 </td>
     <td>68.1 </td>
-    <td>68.2 </td>
     <td>26.4 </td>
     <td>64.6 </td>
   </tr>
@@ -107,16 +124,20 @@ datasets:
     <td nowrap="nowrap" align="left" ><b>OmniLMM-12B</b></td>
     <td align="right">11.6B </td>
     <td>1637 </td>
+    <td>71.6 </td>
     <td>40.7 </td>
     <td>3.45 / 68.8 </td>
+    <td>90.3 / 95.5 </td>
     <td>71.1 </td>
-    <td>71.6 </td>
     <td>34.9 </td>
     <td>72.0 </td>
   </tr>
 </tbody>
 </table>
 <small>†: Proprietary models</small>
+<br>
+</details>
+
 
 ## Demo
 Click here to try out the Demo of [OmniLMM-12B](http://120.92.209.146:8081).