cjvt
/

OPT_GaMS-1B-Chat

@@ -8,30 +8,30 @@ pipeline_tag: text-generation
 # Model Card for OPT_GaMS-1B-Chat
-This is the 1B OPT model additionally pretrained on Slovene data. The model was created as a part of project Povejmo: https://www.cjvt.si/povejmo/.
-The model was instruction tuned on Slovene Self-instruct dataset (20.000) examples.
-## Pretraining data
-The model was additionally pretrained on the following Slovene, English, and Croatian-Bosnian-Serbian (CBS) corpora:
-| Corpus | Language | # Tokens | Percentage |
-| :----- | :------- | :------: | :--------: |
-| Metafida | Slovene | 6.59 B | 13.89 % |
-| KAS | Slovene | 3.61 B | 7.62 % |
-| Trendi | Slovene | 1.4 B | 2.96 % |
-| mC4 | Slovene | 5.5 B | 11.6 % |
-| MaCoCu | Slovene | 4.68 B | 9.86 % |
-| CC100 | Slovene | 0.54 B | 1.14 % |
-| Rižnica | Croatian | 0.21 B | 0.44 % |
-| Hr News | Croatian | 4.16 B | 8.77 % |
-| MaCoCu HBS | CBS | 15.65 B | 32.98 % |
-| Wikipedia | English | 4.7 B | 9.9 % |
-| CC-News | English | 0.4 B | 0.83 % |
-The total size of additional training data is **28.13 B** tokens.
-## Model usage
 The inference can be done using the following snippet of code:
@@ -58,191 +58,84 @@ response = pipeline(new_message, max_length=500)
 print("Model's response:", response[0]["generated_text"][-1]["content"])
 ```
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
 ### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 # Model Card for OPT_GaMS-1B-Chat
+We proudly present the familly of GaMS (Generative Model for Slovene) models. The 1B version is based on [Facebook's OPT model](https://huggingface.co/facebook/opt-1.3b) and is adapted for Slovene. OPT_GaMS models use original OPT tokenizer. This is the instruction-tuned version of the model.
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/652d40a78fa1fbb0aae165bb/YnvcP1x7CHH-eTY2oB-69.png)
+## Acknowledgment
+The model was developed within the [PoVeJMo](https://povejmo.si) research program (Adaptive Natural Language Processing with Large Language Models}; Prilagodljiva obdelava naravnega jezika s pomočjo velikih jezikovnih modelov), particularly within the research project titled SloLLaMai -- Open-access computationally efficient models for Slovenian, funded within the Recovery and Resilience Plan (NOO; Načrt za okrevanje in odpornost) by the Slovenian Research and Innovation Agency (ARIS) and NextGenerationEU. The authors also acknowledge the financial support from the Slovenian Research and Innovation Agency (research core funding No. P6-0411 -- Language Resources and Technologies for Slovene).
+We thank everyone, who worked on data collection and preparation, enabling us to train our model. Special thanks goes to: Nikola Ljubešić, Tjaša Arčon, Jaka Čibej, Simon Krek, Tomaž Erjavec and Iztok Kosem.
+## Basic information
+- **Developed by:** team of researchers at University of Ljubljana, Faculty for Computer and Information Science and XLAB.doo. Team members: Domen Vreš, Martin Božič, Aljaž Potočnik, Tomaž Martinčič, Iztok Lebar Bajec, Timotej Petrič and Marko Robnik-Šikonja.
+- **Language:** Slovene
+- **License:** Apache 2.0
+- **Repository:** https://github.com/SloLama/NeMo
+- **Paper:** https://www.sdjt.si/wp/wp-content/uploads/2024/09/JT-DH-2024_Vres_Bozic_Potocnik_Martincic_Robnik.pdf
+## Intended usage
+This version of the model is quite small and lacks safety tuning. Hence, using it as a general purpose model is **STRONGLY DISCOURAGED!!!** The model might also contain certain biases. We do not recommend usage of this model in any other language than Slovene.
+The model can be efficiently tuned for specific use cases as suggested by promising results of fine-tuned models on SuperGLUE and SI-NLI benchmarks
+## How to get started with the model
 The inference can be done using the following snippet of code:
 print("Model's response:", response[0]["generated_text"][-1]["content"])
 ```
+## Training details
+### Training data
+The model was additionally pretrained on the following Slovene, English, and Croatian-Bosnian-Serbian (CBS) corpora:
+| Corpus | Language | # Tokens | Percentage |
+| :----- | :------- | :------: | :--------: |
+| Metafida | Slovene | 6.59 B | 13.89 % |
+| KAS | Slovene | 3.61 B | 7.62 % |
+| Trendi | Slovene | 1.4 B | 2.96 % |
+| mC4 | Slovene | 5.5 B | 11.6 % |
+| MaCoCu | Slovene | 4.68 B | 9.86 % |
+| CC100 | Slovene | 0.54 B | 1.14 % |
+| Rižnica | Croatian | 0.21 B | 0.44 % |
+| Hr News | Croatian | 4.16 B | 8.77 % |
+| MaCoCu HBS | CBS | 15.65 B | 32.98 % |
+| Wikipedia | English | 4.7 B | 9.9 % |
+| CC-News | English | 0.4 B | 0.83 % |
+The total size of additional training data is **47.44 B** tokens.
 ### Training Procedure
+The model was trained using NeMo framework on Slovene HPC Vega, utilizing 64 A100 GPUs at once. Training took approximately 16 hours. The model was trained with batch size 1024 (2 million tokens) using Adam optimizer and cosine learning rate scheduler with 1000 warmup and constant steps.
+### Supervised Finetuning (SFT)
+The model was trained on [GaMS-Instruct](http://hdl.handle.net/11356/1971) dataset (20.000 examples). The currated version of the dataset (7.000 examples) is publicly available. 19.050 examples were used as a training set and 950 examples were used as a validation set.
+The model was LoRA tuned on 7 epochs with rank 1024. The model was trained with batch size 64 using Adam optimizer and cosine learning rate scheduler with 300 warmup steps.
 ## Evaluation
+The model was evaluated using [Slovene SuperGLUE](https://slobench.cjvt.si/leaderboard/view/3) and [SI-NLI](https://slobench.cjvt.si/leaderboard/view/9) tasks on [SloBench](https://slobench.cjvt.si). Additionally, the models was evaluated on imporved version of Slovenian-LLM-eval introduced by Aleksa Gordić. All decoder-type models were evaluated using few-shot prompts and were not finetuned on the benchmark (except for the versions with finetuned in the name).
+### SuperGLUE results
+| Model | SuperGLUE Average | BoolQ Accuracy | CB Accuracy | CB F1 Score | CB Average |	COPA Accuracy |	MultiRC EM | MultiRC F1a Score | MultiRC Average | RTE Accuracy | WSC Accuracy |
+| :---- | :---------------: | :------------: | :---------: | :---------: | :--------: | :-----------: | :--------: | :---------------: | :-------------: | :----------: | :----------: |
+| OPT_GaMS-1B                | 0.4408     | 0.5667     | 0.5040     | 0.3885     | 0.4463     | 0.5020     | 0.0961     | 0.2543     | 0.1752     | 0.4138     | 0.5411     |
+| GaMS-1B                    | 0.4604     | 0.5000     | 0.6200     | 0.4565     | 0.5382     | 0.4920     | 0.1351     | 0.2675     | 0.2013     | 0.4828     | 0.5479     |
+| OPT_GaMS-1B-Chat           | 0.4165     | 0.7000     | 0.3720     | 0.2961     | 0.3341     | 0.4600     | 0.1111     | 0.3448     | 0.2280     | 0.4138     | 0.3630     |
+| GaMS-1B-Chat               | 0.4570     | **0.8000** | 0.4880     | 0.3023     | 0.3951     | 0.4840     | 0.1081     | 0.2428     | 0.1755     | 0.5172     | 0.3699     |
+| OPT_GaMS-1B-Chat finetuned | 0.5645     | 0.7000     | 0.8040     | 0.5884     | 0.6962     | 0.5860     | 0.1021     | 0.4808     | 0.2914     | 0.5862     | 0.5274     |
+| GaMS-1B-Chat finetuned     | 0.5806     | 0.7333     | **0.8120** | 0.5592     | 0.6856     | 0.5080     | 0.1381     | 0.4882     | 0.3132     | 0.5862     | **0.6575** |
+| SlovenianGPT-Chat*         | 0.5078     | 0.7333     | 0.3920     | 0.3829     | 0.3874     | **0.6840** | **0.2432** | 0.4944     | **0.3688** | 0.5172     | 0.3562     |
+| CroSloEngual BERT          | **0.6078** | 0.7333     | 0.7920     | **0.7437** | **0.7679** | 0.5720     | 0.0931     | **0.5241** | 0.3086     | **0.6552** | 0.6096     |
+*SlovenianGPT-Chat was obtained by instruction-tuning Aleksa Gordić's [SlovenianGPT](https://huggingface.co/gordicaleksa/SlovenianGPT) on our instruction dataset.
+### SI-NLI results
+| Model | Accuracy | P(entailment) | R(entailment) | F1(entailment) | P(neutral) |	R(neutral) | F1(neutral) |	P(contradiction) |	R(contradiction) |	F1(contradiction) |
+| :---- | :------: | :-----------: | :-----------: | :------------: | :--------: | :---------: | :---------: | :---------------: | :---------------: | :----------------: |
+| OPT_GaMS-1B                | 0.3277     | 0.3407     | 0.6754     | 0.4529     | 0.3538     | 0.1402     | 0.2009     | 0.2632     | 0.1524     | 0.1931     |
+| GaMS-1B                    | 0.3317     | 0.3418     | 0.4327     | 0.3819     | 0.3353     | 0.5122     | 0.4053     | 0.2344     | 0.0457     | 0.0765     |
+| OPT_GaMS-1B-Chat           | 0.3447     | 0.3515     | 0.6784     | 0.4631     | 0.3386     | 0.3293     | 0.3338     | 0.2105     | 0.0122     | 0.0231     |
+| GaMS-1B-Chat               | 0.3417     | 0.3405     | **0.9737** | 0.5045     | 0.2857     | 0.0061     | 0.0119     | 0.4615     | 0.0183     | 0.0352     |
+| OPT_GaMS-1B-Chat finetuned | 0.7244     | 0.7065     | 0.8304     | 0.7634     | 0.7269     | 0.6006     | 0.6578     | 0.7446     | 0.7378     | 0.7412     |
+| GaMS-1B-Chat finetuned     | 0.7144     | 0.8037     | 0.6345     | 0.7092     | 0.7247     | 0.6341     | 0.6764     | 0.6531     | **0.8780** | 0.7490     |
+| SlovenianGPT-Chat*         | 0.4729     | 0.4399     | 0.7281     | 0.5485     | 0.3719     | 0.1372     | 0.2004     | 0.5723     | 0.5427     | 0.5571     |
+| GPT-3.5-Turbo finetuned    | **0.8567** | **0.8464** | 0.8538     | **0.8501** | **0.8041** | **0.8384** | **0.8209** | **0.9260** | **0.8780** | **0.9014** |
+| SloBERTa                   | 0.7375     | 0.8127     | 0.7105     | 0.7582     | 0.6844     | 0.7470     | 0.7143     | 0.7273     | 0.7561     | 0.7414     |
+| CroSloEngual BERT          | 0.6623     | 0.7147     | 0.6667     | 0.6899     | 0.6072     | 0.6646     | 0.6346     | 0.6719     | 0.6555     | 0.6636     |
+*SlovenianGPT-Chat was obtained by instruction-tuning Aleksa Gordić's [SlovenianGPT](https://huggingface.co/gordicaleksa/SlovenianGPT) on our instruction dataset.
+### Slovenian-LLM-eval results
+| Model | ARC-Challenge Accuracy | ARC-Easy Accuracy | BoolQ Accuracy | HellaSwag Accuracy | NQ-Open EM | OpenBookQA Accuracy | PIQA Accuracy | WinoGrande Accuracy |
+| :---- | :--------------------: | :---------------: | :------------: | :----------------: | :--------------: | :-----------------: | :-----------: | :-----------------: |
+| OPT_GaMS-1B        | 0.2227 &plusmn; 0.0122     | 0.436  &plusmn; 0.0102     | 0.378  &plusmn; 0.0085     | 0.3394 &plusmn; 0.0047     | 0.0003 &plusmn; 0.0003     | 0.214 &plusmn; 0.0184     | 0.6083 &plusmn; 0.0114     | 0.5533 &plusmn; 0.014      |
+| GaMS-1B            | 0.2329 &plusmn; 0.0124     | 0.4743 &plusmn; 0.0102     | 0.3813 &plusmn; 0.0085     | 0.3555 &plusmn; 0.0048     | 0.0036 &plusmn; 0.001      | 0.22  &plusmn; 0.0185     | 0.624  &plusmn; 0.0113     | 0.532  &plusmn; 0.014      |
+| OPT_GaMS-1B-Chat   | 0.2355 &plusmn; 0.0124     | 0.3960 &plusmn; 0.0100     | 0.4398 &plusmn; 0.0087     | 0.3459 &plusmn; 0.0047     | 0.0011 &plusmn; 0.0006     | 0.20  &plusmn; 0.0179     | 0.5778 &plusmn; 0.0115     | 0.5359 &plusmn; 0.014      |
+| GaMS-1B-Chat       | 0.2517 &plusmn; 0.0127     | 0.4394 &plusmn; 0.0102     | 0.4502 &plusmn; 0.0087     | 0.3634 &plusmn; 0.0048     | 0      &plusmn; 0          | 0.196 &plusmn; 0.0178     | 0.6115 &plusmn; 0.0114     | 0.5572 &plusmn; 0.014      |
+| YugoGPT            | 0.2961 &plusmn; 0.0133     | 0.4781 &plusmn; 0.0102     | 0.3783 &plusmn; 0.0085     | 0.3890 &plusmn; 0.0047     | 0.0385 &plusmn; 0.0032     | 0.226 &plusmn; 0.0187     | 0.5816 &plusmn; 0.0115     | 0.5588 &plusmn; 0.014      |
+| SlovenianGPT       | **0.3805 &plusmn; 0.0142** | **0.6498 &plusmn; 0.0098** | 0.4523 &plusmn; 0.0087     | **0.4935 &plusmn; 0.0050** | **0.0432 &plusmn; 0.0034** | **0.27  &plusmn; 0.0199** | **0.6937 &plusmn; 0.0108** | **0.644  &plusmn; 0.0135** |
+| SlovenianGPT-Chat* | 0.3567 &plusmn; 0.014      | 0.5901 &plusmn; 0.0101     | **0.4706 &plusmn; 0.0087** | 0.4719 &plusmn; 0.0050     | 0.0003 &plusmn; 0.0003     | **0.27  &plusmn; 0.0199** | 0.6861 &plusmn; 0.0108     | 0.6425 &plusmn; 0.0135     |
+*SlovenianGPT-Chat was obtained by instruction-tuning Aleksa Gordić's [SlovenianGPT](https://huggingface.co/gordicaleksa/SlovenianGPT) on our instruction dataset.
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/652d40a78fa1fbb0aae165bb/_2h977RjIu0nI_IJG_9bL.png)