Readme_en.md

## Vikhr-Nemo-12B-Instruct-R-21-09-24

### Description

**Vikhr-Nemo** is our flagship unimodal LLM (Large Language Model), representing an improved version of [mistralai/Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) developed by the **VikhrModels** team, primarily adapted for Russian and English languages. The training involved several stages, including **SFT** and **SMPO** – our custom variant of DPO, details of which are available in the *"How This Model Was Created"* section.

The model is optimized for a wide range of use cases, including reasoning, summarization, coding, role-playing, and dialogue maintenance. Vikhr-Nemo has capabilities for multilingual generation and high-performance RAG capabilities. It achieves top scores on our instruction and RAG benchmarks, and we believe that for certain tasks (e.g., RAG), it can rival OpenAI's gpt-4o-mini.

All training code is available in our [effective_llm_alignment](https://github.com/VikhrModels/effective_llm_alignment/) repository on GitHub, and the main datasets are available on our [HF profile](https://huggingface.co/Vikhrmodels).

### Features
1. High-quality generation in Russian, English, and several other languages, thanks to the [Grandmaster-PRO-MAX](https://huggingface.co/datasets/Vikhrmodels/GrandMaster-PRO-MAX) dataset and the base model.
2. Support for system prompts to regulate response styles.
3. Up to 128k token context support thanks to the base model.
4. Grounded RAG mode – the model features a special 'documents' role and a mode for identifying relevant document IDs for user queries and using them for responses, inspired by Command-R’s similar capabilities.

### Metrics and Quality Evaluation

The model was evaluated on our open-source Russian-language SbS benchmark [ru-arena-general](https://github.com/VikhrModels/ru_llm_arena) (50 topics with 10 questions each), where gpt-4-1106-preview acted as the judge, and the [RAG benchmark](https://colab.research.google.com/drive/16730rWQ4-yGqWoooLs0Ece_16frmOniP?usp=sharing) based on the [Grounded-RAG-v2](https://huggingface.co/datasets/Vikhrmodels/Grounded-RAG-RU-v2) test set, judged by gpt-4o.

#### Results on Ru-Arena-General

The reference answers, to which models are compared, were generated by gpt-3.5-turbo-0125, hence it has a win rate of 50%.

Only part of the leaderboard is shown here; for more details, check the benchmark repository.

| Model Name                                       | Winrate  | 95% CI             | Average # Tokens |
|--------------------------------------------------|--------|--------------------|------------------|
| gpt-4-1106-preview                               | 90.9   | (-1.3, 1.0)        | 541              |
| gpt-4o-mini                                      | 83.9   | (-1.8, 1.1)        | 448              |
| **vikhr-nemo-12b-instruct-r-21-09-24**           | **79.8**   | (-2.2, 1.9)        | **627**          |
| gemma-2-9b-it-sppo-iter3                         | 73.6   | (-1.6, 2.2)        | 509              |
| gemma-2-9b-it                                    | 69.2   | (-2.5, 1.9)        | 459              |
| t-lite-instruct-0.1                              | 64.7   | (-2.1, 1.7)        | 810              |
| vikhr-llama3.1-8b-instruct-r-21-09-24            | 63.4   | (-2.1, 2.5)        | 618              |
| suzume-llama-3-8B-multilingual-orpo-borda-half   | 57.1   | (-1.9, 2.2)        | 682              |
| mistral-nemo-instruct-2407                       | 50.5   | (-2.7, 2.6)        | 403              |
| gpt-3.5-turbo-0125                               | 50.0   | (0.0, 0.0)         | 220              |
| c4ai-command-r-v01                               | 49.0   | (-1.7, 2.2)        | 529              |
| meta-llama-3.1-8b-instruct                       | 43.1   | (-2.8, 2.3)        | 628              |

#### RAG Benchmark Results

The test set comprises 200 examples: 100 in-domain questions and 100 out-of-domain questions.

For evaluation, the judge model gpt-4o was instructed to consider relevance and factual completeness based on documents and the reference answer from gpt-4-1106-preview.

For prompt details and evaluations, refer to the [Colab notebook](https://colab.research.google.com/drive/16730rWQ4-yGqWoooLs0Ece_16frmOniP?usp=sharing).

**In-Domain**: Questions related to the provided documents.  
**Out-of-Domain**: Questions deliberately unrelated to the provided documents.

[Table representations of the results follow in the original text.]

<table>
<thead>
  <tr>
    <th rowspan="2">question_type</th>
    <th colspan="3">gpt-4o</th>
  </tr>
  <tr>
    <th>judge_correct_percent</th>
    <th>avg_answer_match_rougeL</th>
    <th>avg_abs_indexes_diff</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>in_domain</td>
    <td>73%</td>
    <td>0.34</td>
    <td>NaN</td>
  </tr>
  <tr>
    <td>out_of_domain</td>
    <td>81%</td>
    <td>0.20</td>
    <td>NaN</td>
  </tr>
</tbody>
</table>

<table>
<thead>
  <tr>
    <th style="visibility: hidden;" rowspan="2">question_type</th>
    <th colspan="3">Vikhr-Nemo-12B-Instruct-R-21-09-24</th>
  </tr>
  <tr>
    <th style="visibility: hidden;">judge_correct_percent</th>
    <th style="visibility: hidden;">avg_answer_match_rougeL</th>
    <th style="visibility: hidden;">avg_abs_indexes_diff</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>in_domain</td>
    <td>68%</td>
    <td>0.41</td>
    <td>0</td>
  </tr>
  <tr>
    <td>out_of_domain</td>
    <td>92%</td>
    <td>0.52</td>
    <td>0</td>
  </tr>
</tbody>
</table>

<table>
<thead>
  <tr>
    <th style="visibility: hidden;" rowspan="2">question_type</th>
    <th colspan="3">gpt-4o-mini</th>
  </tr>
  <tr>
    <th style="visibility: hidden;">judge_correct_percent</th>
    <th style="visibility: hidden;">avg_answer_match_rougeL</th>
    <th style="visibility: hidden;">avg_abs_indexes_diff</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>in_domain</td>
    <td>65%</td>
    <td>0.33</td>
    <td>NaN</td>
  </tr>
  <tr>
    <td>out_of_domain</td>
    <td>73%</td>
    <td>0.18</td>
    <td>NaN</td>
  </tr>
</tbody>
</table>

<table>
<thead>
  <tr>
    <th style="visibility: hidden;" rowspan="2">question_type</th>
    <th colspan="3">gpt-3.5-turbo-0125 </th>
  </tr>
  <tr>
    <th style="visibility: hidden;">judge_correct_percent</th>
    <th style="visibility: hidden;">avg_answer_match_rougeL</th>
    <th style="visibility: hidden;">avg_abs_indexes_diff</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>in_domain</td>
    <td>49%</td>
    <td>0.28</td>
    <td>NaN</td>
  </tr>
  <tr>
    <td>out_of_domain</td>
    <td>76%</td>
    <td>0.20</td>
    <td>NaN</td>
  </tr>
</tbody>
</table>

### How This Model Was Created

#### Instructional SFT Part

For the SFT training stage, we prepared a large (150k instructions) synthetic dataset [Vikhrmodels/GrandMaster-PRO-MAX](https://huggingface.co/datasets/Vikhrmodels)

#### Instructional SFT Part

For the SFT stage of model training, we have prepared a large (150k instructions) synthetic instructional dataset [Vikhrmodels/GrandMaster-PRO-MAX](https://huggingface.co/datasets/Vikhrmodels/GrandMaster-PRO-MAX). Its unique feature is the built-in Chain-Of-Thought (CoT), which we collected using a modified prompt for gpt-4-turbo. For more details, please refer to the dataset card.

Additionally, to perform RAG Grounding, we have prepared another synthetic dataset - [Vikhrmodels/Grounded-RAG-RU-v2](https://huggingface.co/datasets/Vikhrmodels/Grounded-RAG-RU-v2) (50k dialogues). The pipeline for its construction is quite complex, so you can find more information in the dataset card.

#### SMPO Alignment Stage

To further improve the quality of responses, we used the following pipeline:
1) Trained a custom Reward model (it will not be released publicly for now).
2) Deduplicated and filtered the original Vikhrmodels/GrandMaster-PRO-MAX dataset using the Reward model, resulting in around 10k of the highest-quality and most diverse dialogues.
3) Performed Rejection Sampling with the SFT checkpoint using the resulting dataset and Reward model. We generated 7 hypotheses and selected the 2 worst ones as rejected.
4) Fine-tuned the SFT checkpoint using our SMPO method with the dataset obtained from step 3. SMPO was designed and chosen as the method to enhance the stability of preference training under Rejection Sampling conditions and to achieve the desired margin.

The implementation of SMPO, rejection sampling, etc., can be found in our [effective_llm_alignment](https://github.com/VikhrModels/effective_llm_alignment/) library on GitHub.

The idea of using SMPO over other PO methods arose from numerous experiments with classical methods and the need for better convergence control. While other methods (e.g., SimPO) can achieve similar results with careful tuning, we aimed to stabilize the process and combine the best practices from other methods.

### How to Work with RAG

The role of "documents" represents a list of dictionaries describing document content, using `json.dumps(array, ensure_ascii=False)` (see example below). \
The document content can be presented in **3** different formats: **Markdown**, **HTML**, **Plain Text**. The content of each document can be a chunk of text up to 4k characters long.

```json
[
  {
    "doc_id": (0..5),
    "title": "(null or str)",
    "content": "(html or markdown or plain text)"
  }
]
```

#### Example of Correct Usage with an OpenAI-like API

Launching the vLLM server: `vllm serve --dtype half --max-model-len 32000 -tp 1 Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24 --api-key token-abc123`

```python
GROUNDED_SYSTEM_PROMPT = "Your task is to answer the user's questions using only the information from the provided documents. Give two answers to each question: one with a list of relevant document identifiers and the second with the answer to the question itself, using documents with these identifiers."

documents = [
  {
    "doc_id": 0,
    "title": "Global Warming: Glaciers",
    "content": "Over the past 50 years, the volume of glaciers worldwide has decreased by 30%."
  },
  {
    "doc_id": 1,
    "title": "Global Warming: Sea Level",
    "content": "The global sea level has risen by 20 cm since 1880 and continues to rise by 3.3 mm per year."
  }
]
sample_history = [
    {'role': 'system', 'content': GROUNDED_SYSTEM_PROMPT}, 
    {'role': 'documents', 'content': json.dumps(documents, ensure_ascii=False)},
    {'role': 'user', 'content': 'Global warming'}
]
relevant_indexes = llm_client.chat.completions.create(
    model=llm_model,
    messages=sample_history,
    temperature=0.0,
    max_tokens=2048
).choices[0].message.content

print('Using documents: ' + relevant_indexes + '\n----')
final_answer = llm_client.chat.completions.create(
    model=llm_model,
    messages=sample_history + [{'role': 'assistant', 'content': relevant_indexes}],
    temperature=0.3,
    max_tokens=2048
).choices[0].message.content

print(final_answer)
```

The response after running the code might look like this:

    Using documents: {"relevant_doc_ids": [0, 1]}
    ----

    Global warming is a long-term increase in the Earth's atmosphere and ocean temperatures. This phenomenon has numerous consequences for our planet, and two key aspects based on the data available to me are:
    
    1. **Reduction of glacier volume**: Over the past 50 years, there has been a significant reduction in glacier volume worldwide. According to the data, glacier volume has decreased by 30%. This could be related to glacier melting due to rising temperatures, which is one of the signs of global warming.
    
    2. **Sea level rise**: The global sea level is also rising, which is associated with the melting of glaciers and ice sheets, as well as the expansion of water as it warms. Since 1880, the sea level has risen by 20 centimeters, and this process continues, with an annual increase of 3.3 millimeters.
    
    These changes have serious consequences for ecosystems, climate, and human society. Glacier melting leads to sea-level rise, which can result in the flooding of coastal areas and islands, as well as changes in water resources and climate patterns.

Using the model's first response, `relevant_indexes` (JSON), one can determine whether the model found information in the documents. The model is trained to return an empty array if no information is found, in which case it will state that it couldn’t find the information in the knowledge base (when generating the second response).

### Nuances and Limitations
- The model has a **low level of response safety** and is focused on correctly and fully executing instructions. Keep this in mind during usage and test it independently. This can be partially corrected with system prompts and additional user prompt guidance about the importance of safety.
- System prompts are not intended for character descriptions; we recommend using them to specify the response style (e.g., "answer only in JSON format"). Additionally, it’s preferable to write them **in English**, as this was the case in the dataset; using English in system prompts does not affect the response language.
- The RAG mode **requires the presence** of the system prompt `GROUNDED_SYSTEM_PROMPT` described in the "How to Work with RAG" section. The model may sometimes add general knowledge information to the response along with the information present in the documents.
- The model works best with low temperatures (0.1-0.5) and top_k (30-50). At a temperature of 1.0, random generation defects were observed.

### Authors 
- Sergei Bratchikov, [NLP Wanderer](https://t.me/nlpwanderer), Vikhr Team
- Konstantin Korolev, Vikhr Team
- Aleksandr Nikolich, Vikhr Team

### Cite
```
@inproceedings{nikolich2024vikhr,
  title={Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for Russian},
  author={Aleksandr Nikolich and Konstantin Korolev and Sergei Bratchikov and Igor Kiselev and Artem Shelmanov},
  booktitle={Proceedings of the 4th Multilingual Representation Learning (MRL) Workshop @ EMNLP-2024},
  year={2024},
  organization={Association for Computational Linguistics}
}
```