fgaim commited on
Commit
6fe1195
1 Parent(s): 6be34e6

Update README

Browse files
Files changed (1) hide show
  1. README.md +17 -3
README.md CHANGED
@@ -6,16 +6,30 @@ widget:
6
 
7
  # BERT Base for Tigrinya Language
8
 
9
- We pretrain a BERT base-uncased model for Tigrinya on a dataset of 40 million tokens trained for 40 epochs.
10
 
11
- Contained in this repo is the original pretrained Flax model that was trained on a TPU v3.8 and it's corresponding PyTorch version.
12
 
13
  ## Hyperparameters
14
 
15
- The hyperparameters corresponding to model sizes mentioned above are as follows:
16
 
17
  | Model Size | L | AH | HS | FFN | P | Seq |
18
  |------------|----|----|-----|------|------|------|
19
  | BASE | 12 | 12 | 768 | 3072 | 110M | 512 |
20
 
21
  (L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
  # BERT Base for Tigrinya Language
8
 
9
+ We pre-train a BERT base-uncased model for Tigrinya on a dataset of 40 million tokens trained for 40 epochs.
10
 
11
+ This repo contains the original pre-trained Flax model that was trained on a TPU v3.8 and its corresponding PyTorch version.
12
 
13
  ## Hyperparameters
14
 
15
+ The hyperparameters corresponding to the model sizes mentioned above are as follows:
16
 
17
  | Model Size | L | AH | HS | FFN | P | Seq |
18
  |------------|----|----|-----|------|------|------|
19
  | BASE | 12 | 12 | 768 | 3072 | 110M | 512 |
20
 
21
  (L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)
22
+
23
+
24
+ ## Citation
25
+
26
+ If you use this model in your product or research, please cite as follows:
27
+
28
+ ```
29
+ @article{Fitsum2021TiPLMs,
30
+ author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
31
+ title={Monolingual Pre-trained Language Models for Tigrinya},
32
+ year=2021,
33
+ publisher={WiNLP 2021 at EMNLP 2021}
34
+ }
35
+ ```