Pieter Delobelle commited on
Commit
1751a3e
1 Parent(s): 86c091d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -5
README.md CHANGED
@@ -26,7 +26,7 @@ widget:
26
  # About RobBERTje
27
  RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.
28
 
29
- We are also continuously working on releasing better-performing models, so watch this page for updates.
30
 
31
  # News
32
  - **July 2, 2021**: Publicly released 4 RobBERTje models.
@@ -35,14 +35,15 @@ We are also continuously working on releasing better-performing models, so watch
35
  # The models
36
  | Model | Description | Parameters | Training size | Huggingface id |
37
  |--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
38
- | Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-non-shuffled](https://huggingface.co/DTAI-KULeuven/) |
39
- | Shuffled | Trained on the publicly available and shuffled OSCAR corpus. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/) |
40
- | Merged (p=0.5) | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/) |
41
- | BORT | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). | 46 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/) |
42
 
43
  # Results
44
 
45
  ## Intrinsic results
 
46
  We calculated the _pseudo perplexity_ (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.
47
  | Model | PPPL |
48
  |-------------------|-----------|
 
26
  # About RobBERTje
27
  RobBERTje is a collection of distilled models based on [RobBERT](http://github.com/iPieter/robbert). There are multiple models with different sizes and different training settings, which you can choose for your use-case.
28
 
29
+ We are also continuously working on releasing better-performing models, so watch [the repository](http://github.com/iPieter/robbertje) for updates.
30
 
31
  # News
32
  - **July 2, 2021**: Publicly released 4 RobBERTje models.
 
35
  # The models
36
  | Model | Description | Parameters | Training size | Huggingface id |
37
  |--------------|-------------|------------------|-------------------|------------------------------------------------------------------------------------|
38
+ | Non-shuffled | Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation. | 74 M | 1 GB | this model |
39
+ | Shuffled | Trained on the publicly available and shuffled OSCAR corpus. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-shuffled](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-shuffled) |
40
+ | Merged (p=0.5) | Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%. | 74 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-merged](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-merged) |
41
+ | BORT | A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). | 46 M | 1 GB | [DTAI-KULeuven/robbertje-1-gb-bort](https://huggingface.co/DTAI-KULeuven/robbertje-1-gb-bort) |
42
 
43
  # Results
44
 
45
  ## Intrinsic results
46
+
47
  We calculated the _pseudo perplexity_ (PPPL) from [cite](), which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.
48
  | Model | PPPL |
49
  |-------------------|-----------|