---
library_name: peft
license: cc-by-nc-2.0
language:
- fr
- en
tags:
- pytorch
- llama
- code
---

## Aria 7B V3

We decided to build a V3 of Aria 7B based on Mistral instruct instead of LLAMA 2. The base model has been quantized with Qlora to reduce the model size and trained on a high quality french dataset.

## Base Model : Mistral-7B-Instruct-v0.1

## Technical issues Fixed & Limits of base model

We noticed that the base model had a common issue of mixing french and english when the request was done in french in some cases,not all of them. This issue was more visible for
prompts over 1000 tokens. By training the base model on our dataset, we fixed this issue and allow the model to reply in the same specific language used for the question to answer.
This pain-point is a valuable upgrade for corporate users in non-english areas willing to deploy a model with an increased quality and accuracy in french language.

## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
### Framework versions


- PEFT 0.5.0