Behnamm commited on
Commit
f97bde5
1 Parent(s): 3590807

Update src/about.py

Browse files
Files changed (1) hide show
  1. src/about.py +8 -4
src/about.py CHANGED
@@ -42,6 +42,8 @@ linguistic skills and their level of bias, ethics, and trustworthiness. **These
42
  explaining the data and performance of relevent models.**
43
 
44
  Note: **We plan to release an evaluation framework soon in which the details and methods of evaluation are specified.**
 
 
45
  """
46
 
47
  # Which evaluations are you running? how can people reproduce what you have?
@@ -51,10 +53,10 @@ For now, the only competitive open language models capable of properly speaking
51
  There are only a few capable multilingual LLMs in Persian that derive their main knowledge from English. A Persian LLM is almost an imagination right now as there doesn't exist
52
  that many models being expert in Persian in the first place.
53
 
54
- Our goals are to provide a benchmark on diverse domains and tasks that provides insights on how much is the gap between the SOTA models right now in different grounds.
55
  This benchmark can also be used by multilingual researchers to measure how well their model performs in a language like Persian.
56
 
57
- We use our own framework to evaluate the models on the following benchmarks (TO BE RELEASED SOON)
58
  ### Tasks
59
  - PeKA: Persian Knowledge Assesment (0-shot) - a set of multiple-choice questions that tests the level of native knowledge in Persian language in more 15 domains and categories: From art to history and geography, cinema, tv, sports, law and medicine, and much more.
60
  - PersBETS: Persian Bias Ethics Toxicity and Skills (0-shot) - a test of model's capability in linguistic skills such as Grammar and Praphrasing, and also questions examining the bias, ethics, and toxicity of the model.
@@ -64,13 +66,15 @@ We use our own framework to evaluate the models on the following benchmarks (TO
64
  - <a href="https://arxiv.org/abs/2012.06154" target="_blank"> ParsiNLU QQP </a> (max[0,2,5,10]-shot) - task of deciding whether a whether two given questions are paraphrases of each other or not.
65
 
66
  For all these evaluations, a higher score is a better score.
 
67
  We chose these benchmarks for now, but several other benchmarks are going to be added later to help us perform a more thorough examination of models.
 
68
  The last two benchmarks, ParsiNLU NLI and ParsiNLU QQP are evaluated in different few-shot settings and then the maximum score is returned as the final evaluation.
69
- We argue that is indeed a fair evaluation method since many ,ight-weight models (around ~7B and less) can have a pooor in-context learning and thus they perform better
70
  in small shots. We wish to not hold this against the model by trying to measure performances in different settings and take the maximum score achieved .
71
 
72
  ## REPRODUCIBILITY
73
- (TO BE COMPLETED)
74
  """
75
 
76
  EVALUATION_QUEUE_TEXT = """
 
42
  explaining the data and performance of relevent models.**
43
 
44
  Note: **We plan to release an evaluation framework soon in which the details and methods of evaluation are specified.**
45
+
46
+ Note: The two closed benchmarks are supposed to change drastically in the next two weeks, so this leaderboard's numeric values may need modifications once the framework and the corresponding data are made open.
47
  """
48
 
49
  # Which evaluations are you running? how can people reproduce what you have?
 
53
  There are only a few capable multilingual LLMs in Persian that derive their main knowledge from English. A Persian LLM is almost an imagination right now as there doesn't exist
54
  that many models being expert in Persian in the first place.
55
 
56
+ Our goal is to provide a benchmark on diverse domains and tasks that provide insights on how much is the gap between current Persian LLMs and the SOTA multilingual models right now in different grounds.
57
  This benchmark can also be used by multilingual researchers to measure how well their model performs in a language like Persian.
58
 
59
+ We use our own framework to evaluate the models on the following benchmarks (TO BE RELEASED SOON).
60
  ### Tasks
61
  - PeKA: Persian Knowledge Assesment (0-shot) - a set of multiple-choice questions that tests the level of native knowledge in Persian language in more 15 domains and categories: From art to history and geography, cinema, tv, sports, law and medicine, and much more.
62
  - PersBETS: Persian Bias Ethics Toxicity and Skills (0-shot) - a test of model's capability in linguistic skills such as Grammar and Praphrasing, and also questions examining the bias, ethics, and toxicity of the model.
 
66
  - <a href="https://arxiv.org/abs/2012.06154" target="_blank"> ParsiNLU QQP </a> (max[0,2,5,10]-shot) - task of deciding whether a whether two given questions are paraphrases of each other or not.
67
 
68
  For all these evaluations, a higher score is a better score.
69
+
70
  We chose these benchmarks for now, but several other benchmarks are going to be added later to help us perform a more thorough examination of models.
71
+
72
  The last two benchmarks, ParsiNLU NLI and ParsiNLU QQP are evaluated in different few-shot settings and then the maximum score is returned as the final evaluation.
73
+ We argue that is indeed a fair evaluation method since many light-weight models (around ~7B and less) can have a pooor in-context learning and thus they perform better
74
  in small shots. We wish to not hold this against the model by trying to measure performances in different settings and take the maximum score achieved .
75
 
76
  ## REPRODUCIBILITY
77
+ The parameters used for evaluation along with instructions and prompts will be available once the framework is release. (TO BE COMPLETED)
78
  """
79
 
80
  EVALUATION_QUEUE_TEXT = """