Update README.md

README.md

@@ -1,82 +1,259 @@
 ---
+license: apache-2.0
+datasets:
+- FreedomIntelligence/ApolloMoEDataset
 language:
+- ar
 - en
-pipeline_tag: text-generation
+- zh
+- ko
+- ja
+- mn
+- th
+- vi
+- lo
+- mg
+- de
+- pt
+- es
+- fr
+- ru
+- it
+- hr
+- gl
+- cs
+- co
+- la
+- uk
+- bs
+- bg
+- eo
+- sq
+- da
+- sa
+- 'no'
+- gn
+- sr
+- sk
+- gd
+- lb
+- hi
+- ku
+- mt
+- he
+- ln
+- bm
+- sw
+- ig
+- rw
+- ha
+metrics:
+- accuracy
+base_model:
+- Qwen/Qwen2-1.5B
+pipeline_tag: question-answering
 tags:
-- pretrained
-license: apache-2.0
+- biology
+- medical
 ---
+# Democratizing Medical LLMs For Much More Languages
 
-# Qwen2-1.5B
+Covering 12 Major Languages including English, Chinese, French, Hindi, Spanish, Arabic, Russian, Japanese, Korean, German, Italian, Portuguese and 38 Minor Languages So far.
+<center>
 
-## Introduction
 
-Qwen2 is the new series of Qwen large language models. For Qwen2, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters, including a Mixture-of-Experts model. This repo contains the 1.5B Qwen2 base language model.
 
-Compared with the state-of-the-art opensource language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most opensource models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.
+<p align="center">
+   📃 <a href="https://arxiv.org/abs/2410.10626" target="_blank">Paper</a> • 🌐 <a href="" target="_blank">Demo</a> • 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset" target="_blank">ApolloMoEDataset</a> • 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench" target="_blank">ApolloMoEBench</a>  • 🤗 <a href="https://huggingface.co/collections/FreedomIntelligence/apollomoe-and-apollo2-670ddebe3bb1ba1aebabbf2c" target="_blank">Models</a> • 🌐 <a href="https://github.com/FreedomIntelligence/Apollo" target="_blank">Apollo</a> • 🌐 <a href="https://github.com/FreedomIntelligence/ApolloMoE" target="_blank">ApolloMoE</a>
+</p>
 
-For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2/), [GitHub](https://github.com/QwenLM/Qwen2), and [Documentation](https://qwen.readthedocs.io/en/latest/).
-<br>
 
 
-## Model Details
-Qwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.
+![Apollo](assets/apollo_medium_final.png)
 
-## Requirements
-The code of Qwen2 has been in the latest Hugging face transformers and we advise you to install `transformers>=4.37.0`, or you might encounter the following error:
-```
-KeyError: 'qwen2'
-```
 
+## 🌈 Update
+
+* **[2024.10.15]** ApolloMoE repo is published！🎉
+
+
+## Architecture
+
+<details>
+  <summary>Click to view the MoE routing image</summary>
+
+  ![ApolloMoE](/assets/hybrid_routing.png)
+
+</details>
+
+## Results
+
+### Dense
+   🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-0.5B" target="_blank">Apollo2-0.5B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-1.5B" target="_blank">Apollo2-1.5B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-2B" target="_blank">Apollo2-2B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-3.8B" target="_blank">Apollo2-3.8B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-7B" target="_blank">Apollo2-7B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo2-9B" target="_blank">Apollo2-9B</a>  
+
+<details>
+  <summary>Click to view the Dense Models Results</summary>
+
+   ![ApolloMoE](assets/dense_results.png)
 
-## Usage
+</details>
 
-We do not advise you to use base language models for text generation. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.
+### Post-MoE
+   🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-0.5B" target="_blank">Apollo-MoE-0.5B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-1.5B" target="_blank">Apollo-MoE-1.5B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-7B" target="_blank">Apollo-MoE-7B</a>  
 
-## Performance
+<details>
+  <summary>Click to view the Post-MoE Models Results</summary>
 
-The evaluation of base models mainly focuses on the model performance of natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, multilingual capability, etc. 
+   ![ApolloMoE](assets/post_moe_results.png)
 
-The datasets for evaluation include: 
- 
-**English Tasks**: MMLU (5-shot), MMLU-Pro (5-shot), GPQA (5shot), Theorem QA (5-shot), BBH (3-shot), HellaSwag (10-shot), Winogrande (5-shot), TruthfulQA (0-shot), ARC-C (25-shot)
- 
-**Coding Tasks**: EvalPlus (0-shot) (HumanEval, MBPP, HumanEval+, MBPP+), MultiPL-E (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript)
+</details>
+
+   
+
+
+​      
+
+
+## Usage Format
+#### Apollo2
+- 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>
+- 2B, 9B: User:{query}\nAssistant:{response}\<eos\>
+- 3.8B: <|user|>\n{query}<|end|><|assisitant|>\n{response}<|end|>
+
+#### Apollo-MoE
+- 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>
   
-**Math Tasks**: GSM8K (4-shot), MATH (4-shot)
- 
-**Chinese Tasks**: C-Eval(5-shot), CMMLU (5-shot)
- 
-**Multilingual Tasks**: Multi-Exam (M3Exam 5-shot, IndoMMLU 3-shot, ruMMLU 5-shot, mMMLU 5-shot), Multi-Understanding (BELEBELE 5-shot, XCOPA 5-shot, XWinograd 5-shot, XStoryCloze 0-shot, PAWS-X 5-shot), Multi-Mathematics (MGSM 8-shot), Multi-Translation (Flores-101 5-shot)
- 
-
-#### Qwen2-0.5B & Qwen2-1.5B performances
-|  Datasets  |  Phi-2 |   Gemma-2B | MiniCPM |  Qwen1.5-1.8B  |   Qwen2-0.5B  |  Qwen2-1.5B  |
-| :--------| :---------: | :------------: | :------------: |:------------: | :------------: | :------------: |
-|#Non-Emb Params | 2.5B | 2.0B | 2.4B | 1.3B | 0.35B | 1.3B |
-|MMLU | 52.7 | 42.3 | 53.5 | 46.8 | 45.4 | **56.5** |
-|MMLU-Pro | - | 15.9 | - | - | 14.7 | 21.8 |
-|Theorem QA | - | - | - |- | 8.9 | **15.0** |
-|HumanEval | 47.6 |  22.0 |**50.0**| 20.1 | 22.0 | 31.1 |
-|MBPP | **55.0** | 29.2 | 47.3 | 18.0 | 22.0 | 37.4  |
-|GSM8K | 57.2 |  17.7  | 53.8 | 38.4 | 36.5 | **58.5** |
-|MATH  | 3.5 |  11.8  | 10.2 | 10.1 | 10.7 | **21.7** |
-|BBH  | **43.4** |  35.2 | 36.9 | 24.2 | 28.4 | 37.2 |
-|HellaSwag  | **73.1** |  71.4 | 68.3 | 61.4 |  49.3 | 66.6 |
-|Winogrande  | **74.4** |  66.8 | -| 60.3 |  56.8 |  66.2 |
-|ARC-C  | **61.1** |  48.5  | -| 37.9 | 31.5 |  43.9 |
-|TruthfulQA  | 44.5 |  33.1  | -| 39.4 | 39.7 |  **45.9** |
-|C-Eval   | 23.4 |   28.0    | 51.1| 59.7 |  58.2 |  **70.6** |
-|CMMLU   | 24.2 |   -    | 51.1 | 57.8 | 55.1 | **70.3** |
+## Dataset & Evaluation
+
+- Dataset
+  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset" target="_blank">ApolloMoEDataset</a>
+
+   <details><summary>Click to expand</summary>
+
+    ![ApolloMoE](assets/Dataset.png)
+
+    - [Data category](https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus/tree/main/train)
+
+
+   </details>
+
+- Evaluation
+  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench" target="_blank">ApolloMoEBench</a> 
+
+   <details><summary>Click to expand</summary>
   
+     - EN:
+       - [MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options) 
+       - [MedMCQA](https://huggingface.co/datasets/medmcqa/viewer/default/test)
+       - [PubMedQA](https://huggingface.co/datasets/pubmed_qa): Because the results fluctuated too much, they were not used in the paper.
+       - [MMLU-Medical](https://huggingface.co/datasets/cais/mmlu)
+         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
+     - ZH:
+       - [MedQA-MCMLE](https://huggingface.co/datasets/bigbio/med_qa/viewer/med_qa_zh_4options_bigbio_qa/test)
+       - [CMB-single](https://huggingface.co/datasets/FreedomIntelligence/CMB): Not used in the paper
+         - Randomly sample 2,000 multiple-choice questions with single answer.
+       - [CMMLU-Medical](https://huggingface.co/datasets/haonan-li/cmmlu)
+         - Anatomy, Clinical_knowledge, College_medicine, Genetics, Nutrition, Traditional_chinese_medicine, Virology
+       - [CExam](https://github.com/williamliujl/CMExam): Not used in the paper
+         - Randomly sample 2,000 multiple-choice questions
 
-## Citation
 
-If you find our work helpful, feel free to give us a cite.
+     - ES: [Head_qa](https://huggingface.co/datasets/head_qa)
+     - FR:
+       - [Frenchmedmcqa](https://github.com/qanastek/FrenchMedMCQA)
+       - [MMLU_FR]
+         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
+     - HI: [MMLU_HI](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Hindi)
+        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
+     - AR: [MMLU_AR](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Arabic)
+        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
+     - JA: [IgakuQA](https://github.com/jungokasai/IgakuQA)
+     - KO: [KorMedMCQA](https://huggingface.co/datasets/sean0042/KorMedMCQA)
+     - IT:
+       - [MedExpQA](https://huggingface.co/datasets/HiTZ/MedExpQA)
+       - [MMLU_IT]
+         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
+     - DE: [BioInstructQA](https://huggingface.co/datasets/BioMistral/BioInstructQA): German part
+     - PT: [BioInstructQA](https://huggingface.co/datasets/BioMistral/BioInstructQA): Portuguese part
+     - RU: [RuMedBench](https://github.com/sb-ai-lab/MedBench)
+
+
+​      
+​      
+
+
+   </details>
+
+
+## Results reproduction
+   <details><summary>Click to expand</summary>
+
+
+   We take Gemma-2b as example
+   1. Download Dataset for project:
+
+      ```
+      bash 0.download_data.sh
+      ```
+    
+   2. Prepare test and dev for specific model:
+
+      
+      - Create test data for with special token, you can use ./util/check.ipynb to check models' special tokens
+        
+       ```
+       bash 1.data_process_test&dev.sh
+       ```
+    
+   3. Prepare train data for specific model (Create tokenized data in advance):
+
+
+      - You can adjust data Training order and Training Epoch in this step
+    
+       ```
+       bash 2.data_process_train.sh
+       ```
+
+   4. Train the model
+
+
+      - If you want to train in Multi Nodes please refer to ./scripts/multi_node_train_*.sh
+
+
+
+
+       ```
+       bash 3.single_node_train_gemma.sh
+       ```
+
+
+   5. Evaluate your model: Generate score for benchmark
+      
+         ```
+         bash 4.eval.sh
+         ```
+
+   6. Evaluate your model: Play with your ckpts in bash
+      
+         ```
+         python ./src/evaluate/cli_demo.py --model_name='./ckpts/your/path/tfmr'
+         ```
+
+   </details>
+
+
+
+##  Citation
+Please use the following citation if you intend to use our dataset for training or evaluation:
 
 ```
-@article{qwen2,
-  title={Qwen2 Technical Report},
-  year={2024}
+@misc{zheng2024efficientlydemocratizingmedicalllms,
+      title={Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts}, 
+      author={Guorui Zheng and Xidong Wang and Juhao Liang and Nuo Chen and Yuping Zheng and Benyou Wang},
+      year={2024},
+      eprint={2410.10626},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2410.10626}, 
 }
-```
+```