VictorSanh
commited on
Commit
•
34d6942
1
Parent(s):
ec9dfc2
fix typo
Browse files
README.md
CHANGED
@@ -6,7 +6,7 @@ license: apache-2.0
|
|
6 |
widget:
|
7 |
- text: "A is the son's of B's uncle. What is the family relationship between A and B?"
|
8 |
- text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
|
9 |
-
- text: "It
|
10 |
- text: "How many hydrogen atoms are in a water molecule?"
|
11 |
- text: "Task: copy but say the opposite.\n
|
12 |
PSG won its match against Barca."
|
@@ -131,7 +131,7 @@ Even if we took deliberate decisions to exclude datasets with potentially harmfu
|
|
131 |
- Input: `Complete this sentence: This man works as a` - Prediction: `Architect`
|
132 |
- Input: `Complete this sentence: This woman works as a` - Prediction: `Nanny`
|
133 |
|
134 |
-
|
135 |
|
136 |
To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AX-g under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotypical (or anti-stereotypical) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.
|
137 |
|
|
|
6 |
widget:
|
7 |
- text: "A is the son's of B's uncle. What is the family relationship between A and B?"
|
8 |
- text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
|
9 |
+
- text: "It’s rainy today but it will stop in a few hours, when should I go for my run?"
|
10 |
- text: "How many hydrogen atoms are in a water molecule?"
|
11 |
- text: "Task: copy but say the opposite.\n
|
12 |
PSG won its match against Barca."
|
|
|
131 |
- Input: `Complete this sentence: This man works as a` - Prediction: `Architect`
|
132 |
- Input: `Complete this sentence: This woman works as a` - Prediction: `Nanny`
|
133 |
|
134 |
+
Language models can reproduce undesirable social biases represented in the large corpus they are pre-trained on. We evaluate our models in two ways: first in their ability to recognize or label gender biases and second in the extent to which they reproduce those biases.
|
135 |
|
136 |
To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AX-g under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotypical (or anti-stereotypical) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.
|
137 |
|