README.md

---
language:
- en
license: mit
base_model:
- mistralai/Mistral-7B-v0.1
datasets:
- argilla/ultrafeedback-binarized-preferences-cleaned
pipeline_tag: text-generation
model-index:
- name: Mistral-ORPO-β
  results:
  # AI2 Reasoning Challenge (25-Shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
       - type: acc_norm
         name: normalized accuracy
         value: 61.18
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta

  # HellaSwag (10-shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
       - type: acc_norm
         name: normalized accuracy
         value: 84.03
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta

  # TruthfulQA (0-shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
       - type: mc2
         value: 47.69
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta

  # GSM8k (5-shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
       - type: acc
         name: accuracy
         value: 39.8
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta

  # MMLU (5-Shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
       - type: acc
         name: accuracy
         value: 63.26
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta

  # Winogrande (5-shot)
  - task: 
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
       - type: acc
         name: accuracy
         value: 79.24
    source:
      name: Open LLM Leaderboard
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaist-ai%2Fmistral-orpo-beta
  - task:
      type: text-generation
    dataset:
      name: AlpacaEval 1
      type: AlpacaEval
    metrics:
    - type: AlpacaEval 1.0
      value: 91.16%
      name: Win Rate
    source:
      url: https://tatsu-lab.github.io/alpaca_eval/
      name: Leaderboard
  - task:
      type: text-generation
    dataset:
      name: AlpacaEval 2
      type: AlpacaEval
    metrics:
    - type: AlpacaEval 2.0
      value: 12.57%
      name: Win Rate
    source:
      url: https://tatsu-lab.github.io/alpaca_eval/
      name: Leaderboard
  - task:
      type: text-generation
    dataset:
      name: MT-Bench
      type: MT-Bench
    metrics:
    - type: MT-Bench
      value: 7.322
      name: Score
    source:
      url: https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/
      name: self-reported
---
# **Mistral-ORPO-β (7B)**

**Mistral-ORPO** is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using the *[odds ratio preference optimization (ORPO)](https://arxiv.org/abs/2403.07691)*. With ORPO, the model directly learns the preference without the supervised fine-tuning warmup phase. **Mistral-ORPO-β** is fine-tuned exclusively on the 61k instances of the cleaned version of UltraFeedback, [argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned), by [Argilla](https://huggingface.co/argilla).

- **Github Repository**: https://github.com/xfactlab/orpo

## 👍 **Model Performance**

### 1) AlpacaEval & MT-Bench

|Model Name|Size|Align|MT-Bench|AlpacaEval 1.0|AlpacaEval 2.0|
|:--------|:--------------:|:--------------:|:-------------------:|:------------:|:------------:|
|**Mistral-<tt>ORPO</tt>-⍺**|7B|<tt>ORPO</tt>|7.23|87.92|11.33|
|**Mistral-<tt>ORPO</tt>-β**|7B|<tt>ORPO</tt>|7.32|91.41|12.20|
|Zephyr β |7B|DPO|7.34|90.60|10.99|
|TULU-2-DPO |13B|DPO|7.00|89.5|10.12|
|Llama-2-Chat |7B|RLHF|6.27|71.37|4.96|
|Llama-2-Chat |13B|RLHF|6.65|81.09|7.70|

### 2) IFEval

| **Model Type**     | **Prompt-Strict** | **Prompt-Loose** | **Inst-Strict** | **Inst-Loose** |
|--------------------|:-----------------:|:----------------:|:---------------:|:--------------:|
| **Mistral-ORPO-⍺** |       0.5009      |      0.5083      |      0.5995     |     0.6163     |
| **Mistral-ORPO-β** |       0.5287      |      0.5564      |      0.6355     |     0.6619     |

## 🗺️ **MT-Bench by Category**

![image/png](https://cdn-uploads.huggingface.co/production/uploads/6415c043486c7c9a5d151583/1Ifpt0ljCfJPEoZAqlqqy.png)

## 🖥️ **Inference**

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("kaist-ai/mistral-orpo-beta")
tokenizer = AutoTokenizer.from_pretrained("kaist-ai/mistral-orpo-beta")

# Apply chat template
query = [{'role': 'user', 'content': 'Hi! How are you doing?'}]
prompt = tokenizer.apply_chat_template(query, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors='pt')

# Generation with specific configurations
output = model.generate(
  **inputs,
  max_new_tokens=128,
  do_sample=True,
  temperature=0.7
)
response = tokenizer.batch_decode(output)

#<|user|>
#Hi! How are you doing?</s>
#<|assistant|>
#I'm doing well, thank you! How are you?</s>
```

## 📎 **Citation**

```
@misc{hong2024orpo,
      title={ORPO: Monolithic Preference Optimization without Reference Model}, 
      author={Jiwoo Hong and Noah Lee and James Thorne},
      year={2024},
      eprint={2403.07691},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```