norallm
/

normistral-7b-scratch

@@ -94,7 +94,7 @@ We release [our codebase here](https://github.com/ltgoslo/norallm). We compare a
 We use the binary formulation of this task (positive vs. negative).
 <details>
-<summary>Method</summary>
 * Evaluation setting: zero-shot and few-shot perplexity-based evaluation.
 * Prompt: ```"Tekst: {text}\nSentiment:{label}"```, where the ```label``` is either "positiv" or "negativ".
@@ -127,7 +127,7 @@ We use the binary formulation of this task (positive vs. negative).
 [NorQuAD](https://huggingface.co/datasets/ltg/norquad) ([Ivanova et al., 2023](https://aclanthology.org/2023.nodalida-1.17/)) is a dataset for extractive question answering in Norwegian designed similarly to [SQuAD (Rajpurkar et al., 2016)](https://aclanthology.org/D16-1264/).
 <details>
-<summary>Method</summary>
 * Evaluation setting: zero-shot and few-shot settings via natural language generation using the greedy decoding strategy.
 * Prompt: ```"Tittel: {title}\n\nTekst: {text}\n\nSpørsmål: {question}\n\nSvar:{answer}"``` Based on [Brown et al. (2020)](https://arxiv.org/abs/2005.14165).
@@ -154,13 +154,44 @@ We use the binary formulation of this task (positive vs. negative).
 </details>
 ### Machine translation
 [Tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt) [(Tiedemann, 2020)](https://aclanthology.org/2020.wmt-1.139/) is a benchmark for machine translation, which includes hundreds of language pairs. We consider six language pairs (English <-> Bokmål, English <-> Nynorsk, and Bokmål <-> Nynorsk).
 <details>
-<summary>Method</summary>
 * Evaluation setting: zero-shot and few-shot settings via natural language generation using the greedy decoding strategy.
 * Prompt: ```"{source_language}: {source_text}\n{target_language}:{target_text}"```, where the ```source_language``` and ```target_language``` are ```Engelsk```, ```Bokmål```, or ```Nynorsk```. Based on [Garcia et al. (2023)](https://arxiv.org/abs/2302.01398).

 We use the binary formulation of this task (positive vs. negative).
 <details>
+<summary>Method (click to expand)</summary>
 * Evaluation setting: zero-shot and few-shot perplexity-based evaluation.
 * Prompt: ```"Tekst: {text}\nSentiment:{label}"```, where the ```label``` is either "positiv" or "negativ".
 [NorQuAD](https://huggingface.co/datasets/ltg/norquad) ([Ivanova et al., 2023](https://aclanthology.org/2023.nodalida-1.17/)) is a dataset for extractive question answering in Norwegian designed similarly to [SQuAD (Rajpurkar et al., 2016)](https://aclanthology.org/D16-1264/).
 <details>
+<summary>Method (click to expand)</summary>
 * Evaluation setting: zero-shot and few-shot settings via natural language generation using the greedy decoding strategy.
 * Prompt: ```"Tittel: {title}\n\nTekst: {text}\n\nSpørsmål: {question}\n\nSvar:{answer}"``` Based on [Brown et al. (2020)](https://arxiv.org/abs/2005.14165).
 </details>
+### Grammatical error correction
+[ASK-RAW](https://huggingface.co/datasets/ltg/ask-gec) is dataset for Norwegian grammatical error correction (GEC) created by [Matias Jentoft (2023)](https://www.duo.uio.no/handle/10852/103885).
+<details>
+<summary>Method (click to expand)</summary>
+* Evaluation setting: zero-shot and few-shot settings via natural language generation using the greedy decoding strategy.
+* Prompt: ```"Her er eksempler på perfekt korrigering av grammatiske feil:\n\nTekst: {source_text}\nKorreksjon:{target_text}"```
+* Few-shot results show the average scores across 5 repetitions
+* Evaluation script: https://github.com/ltgoslo/norallm/blob/main/initial_evaluation/gec.py
+* Performance metrics: the evaluation metric uses [ERRANT](https://github.com/chrisjbryant/errant/tree/main), which identifies edit-spans and then calculates the F_{0.5} scores between the gold edits and predicted edits.
+</details>
+<details open>
+<summary>Results on [the ASK corpus](https://huggingface.co/datasets/ltg/ask-gec) (ERRANT F_{0.5})</summary>
+|Model|0-shot (F0.5)|1-shot (F0.5)|32-shot (F0.5)|
+|---|---|---|---|
+|NorMistral-7b-warm|**40.8**|41.8|48.5|
+|NorMistral-7b-scratch|22.1|28.8|42.1|
+|NorBLOOM-7b|8.7|24.5|32.0|
+|NB-GPT-J|9.1|28.2|30.6|
+|GPT-Sw3-6.7B|30.5|42.9|**50.6**|
+|GPT-Sw3-6.7B-v2|40.6|**43.4**|49.8|
+|Falcon-7B|10.8|12.4|15.5|
+|Mistral-7B-v0.1|26.0|27.4|30.6|
+</details>
 ### Machine translation
 [Tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt) [(Tiedemann, 2020)](https://aclanthology.org/2020.wmt-1.139/) is a benchmark for machine translation, which includes hundreds of language pairs. We consider six language pairs (English <-> Bokmål, English <-> Nynorsk, and Bokmål <-> Nynorsk).
 <details>
+<summary>Method (click to expand)</summary>
 * Evaluation setting: zero-shot and few-shot settings via natural language generation using the greedy decoding strategy.
 * Prompt: ```"{source_language}: {source_text}\n{target_language}:{target_text}"```, where the ```source_language``` and ```target_language``` are ```Engelsk```, ```Bokmål```, or ```Nynorsk```. Based on [Garcia et al. (2023)](https://arxiv.org/abs/2302.01398).