Pieter Delobelle
commited on
Commit
•
1751a3e
1
Parent(s):
86c091d
Update README.md
Browse files
README.md
CHANGED
@@ -26,7 +26,7 @@ widget:
|
|
26 |
# About RobBERTje
|
27 |
RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.
|
28 |
|
29 |
-
We are also continuously working on releasing better-performing models, so watch
|
30 |
|
31 |
# News
|
32 |
- **July 2, 2021**: Publicly released 4 RobBERTje models.
|
@@ -35,14 +35,15 @@ We are also continuously working on releasing better-performing models, so watch
|
|
35 |
# The models
|
36 |
| Model | Description | Parameters | Training size | Huggingface id |
|
37 |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
|
38 |
-
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation. | 74 M | 1 GB |
|
39 |
-
| Shuffled | Trained on the publicly available and shuffled OSCAR corpus. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/) |
|
40 |
-
| Merged (p=0.5) | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/) |
|
41 |
-
| BORT | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). | 46 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/) |
|
42 |
|
43 |
# Results
|
44 |
|
45 |
## Intrinsic results
|
|
|
46 |
We calculated the _pseudo perplexity_ (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.
|
47 |
| Model | PPPL |
|
48 |
|-------------------|-----------|
|
|
|
26 |
# About RobBERTje
|
27 |
RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.
|
28 |
|
29 |
+
We are also continuously working on releasing better-performing models, so watch [the repository](http://github.com/iPieter/robbertje) for updates.
|
30 |
|
31 |
# News
|
32 |
- **July 2, 2021**: Publicly released 4 RobBERTje models.
|
|
|
35 |
# The models
|
36 |
| Model | Description | Parameters | Training size | Huggingface id |
|
37 |
|--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
|
38 |
+
| Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation. | 74 M | 1 GB | this model |
|
39 |
+
| Shuffled | Trained on the publicly available and shuffled OSCAR corpus. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-shuffled) |
|
40 |
+
| Merged (p=0.5) | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-merged) |
|
41 |
+
| BORT | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). | 46 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-bort) |
|
42 |
|
43 |
# Results
|
44 |
|
45 |
## Intrinsic results
|
46 |
+
|
47 |
We calculated the _pseudo perplexity_ (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.
|
48 |
| Model | PPPL |
|
49 |
|-------------------|-----------|
|