Model Card for zefiro-7b-dpo-ITA

Last Update: 20/02/2024

Zefiro is a DPO fine tuned model for the Italian language based on Zefiro-7b-sft-ita. The model can be used as base model for more specific conversationl tasks for Italian language

Model Details

Zefiro DPO is a porting of the Zephyr model to the italian language using the wonderful recipes from alignment-handbook . It has also taken ispiration and insights from the Llamantino model developed by Università di Bari. For the implementation we combined different approaches from the two models mentioned but also from the wondeful communtity of open source.

Model description

• Model type: A 7B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.
• Language(s) (NLP): Primarily Italian
• License: Apache 2
• Finetuned from model: Zefiro-7b-sft-ita
• Developed by: giux78
• Funded by: Business Operating System

Code

Can be reproduced running this colab.

Computation

It has been trained on one A100 from seeweb.it who sponsered the training. I strongly suggest is one of the cheapest and solid GPUs provider.

Evaluations:

Model	Arc-c	HellaS	MMUL	AVG
Mixtral 7x8	52.8	75.1	70.9	66.26666667
LLama2 70b	49.4	70.9	65.1	61.8
zefiro-dpo-7b	52.69	67.09	50.8	56.86
zefiro-base-7b	51.07	63.47	52.97	55.83666667
zefiro-sft-7b	50.98	62.71	51.96	55.21666667
LLama1 34B	42.9	65.4	49.0	52.43333333

Intended uses & limitations

The model was initially fine-tuned on a filtered and preprocessed version of ultrafeedback-preferences-ITA.

Here's how you can run the model using Transformers from 🤗 :

# Install transformers from source - only needed for versions <= v4.34
# pip install git+https://github.com/huggingface/transformers.git
# pip install accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mii-community/zefiro-7b-dpo-ITA"
model = AutoModelForCausalLM.from_pretrained(model_id)
model.to('cuda')
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")


sys_prompt = "Sei un assistente disponibile, rispettoso e onesto. " \
         "Rispondi sempre nel modo piu' utile possibile, pur essendo sicuro. " \
         "Le risposte non devono includere contenuti dannosi, non etici, razzisti, sessisti, tossici, pericolosi o illegali. " \
         "Assicurati che le tue risposte siano socialmente imparziali e positive. " \
         "Se una domanda non ha senso o non e' coerente con i fatti, spiegane il motivo invece di rispondere in modo non corretto. " \
         "Se non conosci la risposta a una domanda, non condividere informazioni false."

messages = [{ 'content' : sys_prompt, 'role' : 'assistant'}, 
            {'content' : 'Crea una lista su cosa mangiare a pranzo ogni giorno della settimana a pranzo e cena', 'role' : 'user'}]


def generate_text(sys_prompt, user_prompt):
    messages = [{ 'content' : sys_prompt, 'role' : 'assistant'}, 
            {'content' : user_prompt, 'role' : 'user'}]
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
    generated_ids = model.generate(**model_inputs, max_new_tokens=1024)
    return tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]


generate_text(sys_prompt, 'cosa ne pensi della politica italiana?')

Bias, Risks, and Limitations

Zefiro-7b-sft-ITA has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition of the corpus was used to train the base model (mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the Falcon 180B model card for an example of this.

Training Data

We used UltraChat-ITA as training data that is a filtered version of the UltraChat. For translating the dataset we combined different tools and API we are also evaluating the best approach for translating many more datasets. We have seen that the translation phase is critical and can introduce incorrect syntax and semantics.

Summary

Zefiro-7b-beta-ITA-v0.1 is finetuned version of mistral-7b using the zephyr approach for the italian language.

Citation

@misc{tunstall2023zephyr,
      title={Zephyr: Direct Distillation of LM Alignment}, 
      author={Lewis Tunstall and Edward Beeching and Nathan Lambert and Nazneen Rajani and Kashif Rasul and Younes Belkada and Shengyi Huang and Leandro von Werra and Clémentine Fourrier and Nathan Habib and Nathan Sarrazin and Omar Sanseviero and Alexander M. Rush and Thomas Wolf},
      year={2023},
      eprint={2310.16944},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

@misc{basile2023llamantino,
      title={LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language}, 
      author={Pierpaolo Basile and Elio Musacchio and Marco Polignano and Lucia Siciliani and Giuseppe Fiameni and Giovanni Semeraro},
      year={2023},
      eprint={2312.09993},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Model Card Authors

giux78

Model Card Contact

**ale.ercolani@gmail.com