Introduction
MoMo-72B is trained via Supervised Fine-Tuning (SFT) using LoRA, with the QWEN-72B model as its base-model.
Note that we did not exploit any form of weight merge.
For leaderboard submission, the trained weight is realigned for compatibility with llama.
MoMo-72B is trained using Moreh's MoAI platform, which simplifies the training of large-scale models, and AMD's MI250 GPU.
Details
Used Librarys
- torch
- peft
Used Datasets
- Open-Orca/SlimOrca
- No other dataset was used
- No benchmark test set or the training set are used
- data contamination check result
| Model | ARC | MMLU | TruthfulQA | GSM8K |
|---|---|---|---|---|
| V1.4(result < 0.1, %) | TBU | 0.73 | 0.71 | TBU |
Used Environments
- AMD MI250 & MoAI platform
- Please visit https://moreh.io/product for more information about MoAI platform
- Or, contact us directly contact@moreh.io
How to use
# pip install transformers==4.35.2
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("moreh/MoMo-72B-LoRA-V1.4")
model = AutoModelForCausalLM.from_pretrained(
"moreh/MoMo-72B-LoRA-V1.4"
)
- Downloads last month
- 1,241
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.