---
license: llama2
language:
- ro
base_model: meta-llama/Llama-2-7b-hf
datasets:
- uonlp/CulturaX
model-index:
- name: OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14
results:
- task:
type: text-generation
dataset:
name: Romanian_Academic_Benchmarks
type: Romanian_Academic_Benchmarks
metrics:
- name: Average accuracy
type: accuracy
value: 38.03
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_arc_challenge
type: OpenLLM-Ro/ro_arc_challenge
metrics:
- name: Average accuracy
type: accuracy
value: 37.95
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_mmlu
type: OpenLLM-Ro/ro_mmlu
metrics:
- name: Average accuracy
type: accuracy
value: 27.22
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_winogrande
type: OpenLLM-Ro/ro_winogrande
metrics:
- name: Average accuracy
type: accuracy
value: 59.29
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_hellaswag
type: OpenLLM-Ro/ro_hellaswag
metrics:
- name: Average accuracy
type: accuracy
value: 57.22
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_gsm8k
type: OpenLLM-Ro/ro_gsm8k
metrics:
- name: Average accuracy
type: accuracy
value: 2.53
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_truthfulqa
type: OpenLLM-Ro/ro_truthfulqa
metrics:
- name: Average accuracy
type: accuracy
value: 44.00
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary
type: LaRoSeDa_binary
metrics:
- name: Average macro-f1
type: macro-f1
value: 83.25
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass
type: LaRoSeDa_multiclass
metrics:
- name: Average macro-f1
type: macro-f1
value: 61.04
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary_finetuned
type: LaRoSeDa_binary_finetuned
metrics:
- name: Average macro-f1
type: macro-f1
value: 98.97
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass_finetuned
type: LaRoSeDa_multiclass_finetuned
metrics:
- name: Average macro-f1
type: macro-f1
value: 87.72
- task:
type: text-generation
dataset:
name: WMT_EN-RO
type: WMT_EN-RO
metrics:
- name: Average bleu
type: bleu
value: 10.01
- task:
type: text-generation
dataset:
name: WMT_RO-EN
type: WMT_RO-EN
metrics:
- name: Average bleu
type: bleu
value: 13.03
- task:
type: text-generation
dataset:
name: WMT_EN-RO_finetuned
type: WMT_EN-RO_finetuned
metrics:
- name: Average bleu
type: bleu
value: 27.85
- task:
type: text-generation
dataset:
name: WMT_RO-EN_finetuned
type: WMT_RO-EN_finetuned
metrics:
- name: Average bleu
type: bleu
value: 39.30
- task:
type: text-generation
dataset:
name: XQuAD
type: XQuAD
metrics:
- name: Average exact_match
type: exact_match
value: 30.15
- task:
type: text-generation
dataset:
name: XQuAD
type: XQuAD
metrics:
- name: Average f1
type: f1
value: 47.03
- task:
type: text-generation
dataset:
name: XQuAD_finetuned
type: XQuAD_finetuned
metrics:
- name: Average exact_match
type: exact_match
value: 67.06
- task:
type: text-generation
dataset:
name: XQuAD_finetuned
type: XQuAD_finetuned
metrics:
- name: Average f1
type: f1
value: 79.96
- task:
type: text-generation
dataset:
name: STS
type: STS
metrics:
- name: Average spearman
type: spearman
value: 7.89
- task:
type: text-generation
dataset:
name: STS
type: STS
metrics:
- name: Average pearson
type: pearson
value: 7.98
- task:
type: text-generation
dataset:
name: STS_finetuned
type: STS_finetuned
metrics:
- name: Average spearman
type: spearman
value: 71.75
- task:
type: text-generation
dataset:
name: STS_finetuned
type: STS_finetuned
metrics:
- name: Average pearson
type: pearson
value: 71.99
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_arc_challenge
type: OpenLLM-Ro/ro_arc_challenge
metrics:
- name: 0-shot
type: accuracy
value: 35.56
- name: 1-shot
type: accuracy
value: 36.42
- name: 3-shot
type: accuracy
value: 38.56
- name: 5-shot
type: accuracy
value: 38.39
- name: 10-shot
type: accuracy
value: 39.07
- name: 25-shot
type: accuracy
value: 39.67
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_mmlu
type: OpenLLM-Ro/ro_mmlu
metrics:
- name: 0-shot
type: accuracy
value: 25.82
- name: 1-shot
type: accuracy
value: 25.48
- name: 3-shot
type: accuracy
value: 27.61
- name: 5-shot
type: accuracy
value: 29.96
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_winogrande
type: OpenLLM-Ro/ro_winogrande
metrics:
- name: 0-shot
type: accuracy
value: 58.72
- name: 1-shot
type: accuracy
value: 58.88
- name: 3-shot
type: accuracy
value: 60.38
- name: 5-shot
type: accuracy
value: 59.19
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_hellaswag
type: OpenLLM-Ro/ro_hellaswag
metrics:
- name: 0-shot
type: accuracy
value: 55.85
- name: 1-shot
type: accuracy
value: 57.06
- name: 3-shot
type: accuracy
value: 57.52
- name: 5-shot
type: accuracy
value: 57.89
- name: 10-shot
type: accuracy
value: 57.79
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_gsm8k
type: OpenLLM-Ro/ro_gsm8k
metrics:
- name: 1-shot
type: accuracy
value: 0.00
- name: 3-shot
type: accuracy
value: 2.96
- name: 5-shot
type: accuracy
value: 4.62
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary
type: LaRoSeDa_binary
metrics:
- name: 0-shot
type: macro-f1
value: 42.78
- name: 1-shot
type: macro-f1
value: 98.00
- name: 3-shot
type: macro-f1
value: 95.13
- name: 5-shot
type: macro-f1
value: 97.07
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass
type: LaRoSeDa_multiclass
metrics:
- name: 0-shot
type: macro-f1
value: 46.41
- name: 1-shot
type: macro-f1
value: 67.36
- name: 3-shot
type: macro-f1
value: 65.16
- name: 5-shot
type: macro-f1
value: 65.23
- task:
type: text-generation
dataset:
name: WMT_EN-RO
type: WMT_EN-RO
metrics:
- name: 0-shot
type: bleu
value: 4.45
- name: 1-shot
type: bleu
value: 8.61
- name: 3-shot
type: bleu
value: 12.25
- name: 5-shot
type: bleu
value: 14.73
- task:
type: text-generation
dataset:
name: WMT_RO-EN
type: WMT_RO-EN
metrics:
- name: 0-shot
type: bleu
value: 1.29
- name: 1-shot
type: bleu
value: 10.78
- name: 3-shot
type: bleu
value: 16.82
- name: 5-shot
type: bleu
value: 23.24
- task:
type: text-generation
dataset:
name: XQuAD_EM
type: XQuAD_EM
metrics:
- name: 0-shot
type: exact_match
value: 5.29
- name: 1-shot
type: exact_match
value: 33.95
- name: 3-shot
type: exact_match
value: 39.24
- name: 5-shot
type: exact_match
value: 42.10
- task:
type: text-generation
dataset:
name: XQuAD_F1
type: XQuAD_F1
metrics:
- name: 0-shot
type: f1
value: 16.17
- name: 1-shot
type: f1
value: 51.84
- name: 3-shot
type: f1
value: 58.82
- name: 5-shot
type: f1
value: 61.29
- task:
type: text-generation
dataset:
name: STS_Spearman
type: STS_Spearman
metrics:
- name: 1-shot
type: spearman
value: -1.74
- name: 3-shot
type: spearman
value: 15.47
- name: 5-shot
type: spearman
value: 9.93
- task:
type: text-generation
dataset:
name: STS_Pearson
type: STS_Pearson
metrics:
- name: 1-shot
type: pearson
value: -1.40
- name: 3-shot
type: pearson
value: 15.00
- name: 5-shot
type: pearson
value: 10.33
---
# Model Card for Model ID
RoLlama2 is a family of pretrained and fine-tuned generative text models for Romanian. This is the repository for the **foundational 7B model**. Links to other models can be found at the bottom of this page.
## Model Details
### Model Description
OpenLLM represents the first open-source effort to build a LLM specialized for Romanian. OpenLLM-Ro developed and publicly releases a collection of Romanian LLMs, both in the form of foundational model and instruct and chat variants.
- **Developed by:** OpenLLM-Ro
- **Language(s):** Romanian
- **License:** Llama2 Community License Agreement
- **Continual pretrained from model:** [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf)
- **Trained using:** [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX)
### Model Sources
- **Repository:** https://github.com/OpenLLM-Ro/llama-recipes
- **Paper:** https://arxiv.org/abs/2406.18266
## Intended Use
### Intended Use Cases
RoLlama2 is intented for research use in Romanian. Base models can be adapted for a variety of natural language tasks while instruction and chat tuned models are intended for assistant-like chat.
### Out-of-Scope Use
Use in any manner that violates the license, any applicable laws or regluations, use in languages other than Romanian.
## How to Get Started with the Model
Use the code below to get started with the model.
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14")
model = AutoModelForCausalLM.from_pretrained("OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14")
input_text = "Mihai Eminescu a fost "
input_ids = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**input_ids, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))
```
## Academic Benchmarks
Model
Average
ARC
MMLU
Winogrande
Hellaswag
GSM8k
TruthfulQA
Llama-2-7b
37.04
36.05
33.66
57.56
48.00
4.75
42.22
RoLlama2-7b-Base-2024-05-14
38.03
37.95
27.22
59.29
57.22
2.53
44.00
## Downstream Tasks
LaRoSeDa
WMT
Few-shot
Finetuned
Few-shot
Finetuned
Model
Binary (Macro F1)
Multiclass (Macro F1)
Binary (Macro F1)
Multiclass (Macro F1)
EN-RO (Bleu)
RO-EN (Bleu)
EN-RO (Bleu)
RO-EN (Bleu)
Llama-2-7b
93.19
54.11
98.43
87.22
14.90
26.61
24.95
39.09
RoLlama2-7b-Base-2024-05-14
83.25
61.04
98.97
87.72
10.01
13.03
27.85
39.30
XQuAD
STS
Few-shot
Finetuned
Few-shot
Finetuned
Model
(EM)
(F1)
(EM)
(F1)
(Spearman)
(Pearson)
(Spearman)
(Pearson)
Llama-2-7b
38.91
56.82
65.46
79.42
9.08
9.07
79.93
81.08
RoLlama2-7b-Base-2024-05-14
30.15
47.03
67.06
79.96
7.89
7.98
71.75
71.99
## RoLlama2 Model Family
| Model | Link |
|--------------------|:--------:|
|RoLlama2-7b-Base-2024-05-14 | [link](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Base-2024-05-14) |
|RoLlama2-7b-Instruct-2024-05-14 | [link](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct-2024-05-14) |
|*RoLlama2-7b-Instruct-2024-10-09*| [link](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct-2024-10-09) |
|RoLlama2-7b-Instruct-DPO-2024-10-09| [link](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct-DPO-2024-10-09) |
## Citation
```
@misc{masala2024vorbecstiromanecsterecipetrain,
title={"Vorbe\c{s}ti Rom\^ane\c{s}te?" A Recipe to Train Powerful Romanian LLMs with English Instructions},
author={Mihai Masala and Denis C. Ilie-Ablachim and Alexandru Dima and Dragos Corlatescu and Miruna Zavelca and Ovio Olaru and Simina Terian-Dan and Andrei Terian-Dan and Marius Leordeanu and Horia Velicu and Marius Popescu and Mihai Dascalu and Traian Rebedea},
year={2024},
eprint={2406.18266},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2406.18266},
}
```