elmadany commited on
Commit
18507ab
1 Parent(s): d402030

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -1,7 +1,7 @@
1
  # AraT5-base
2
  <img src="https://raw.githubusercontent.com/UBC-NLP/araT5/main/AraT5_logo.jpg" alt="drawing" width="30%" height="30%" align="right"/>
3
 
4
- **AraT5-base** is one of three models described in our [AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation
5
  ](https://arxiv.org/abs/2109.12068). In this paper, we introduce three powerful Arabic-specific text-to-text transformer models trained on large Modern Standard Arabic (MSA) and/or Dialectal Arabic (DA) data. **AraT5** is trained on 248GB of text (29B tokens) of MSA and DA, **AraT5-msa** is trained on 70GB of text (7.1B tokens) from MSA data, and **AraT5-tweet** is trained on 178Gb of text (21.9B tokens) from 1.5B Arabic tweets which contains multiple varieties of dialectical Arabic.
6
 
7
  In addition, we provide the three models on two architectures small and base. For all models, we use a learning rate of 0.01, a batch size of 128 sequences, and a maximum sequence length of 512 whereas AraT5-tweet 128 maximum sequence is used. Hence, the original implementation of T5 in the TensorFlow framework is used to train the models. We train the models for 1M steps.8 Training took ∼ 80 days on 1 on Google Cloud TPU with 8 cores (v3.8) from TensorFlow Research Cloud (TFRC).
 
1
  # AraT5-base
2
  <img src="https://raw.githubusercontent.com/UBC-NLP/araT5/main/AraT5_logo.jpg" alt="drawing" width="30%" height="30%" align="right"/>
3
 
4
+ **AraT5-base** is one of three models described in our [**AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation**
5
  ](https://arxiv.org/abs/2109.12068). In this paper, we introduce three powerful Arabic-specific text-to-text transformer models trained on large Modern Standard Arabic (MSA) and/or Dialectal Arabic (DA) data. **AraT5** is trained on 248GB of text (29B tokens) of MSA and DA, **AraT5-msa** is trained on 70GB of text (7.1B tokens) from MSA data, and **AraT5-tweet** is trained on 178Gb of text (21.9B tokens) from 1.5B Arabic tweets which contains multiple varieties of dialectical Arabic.
6
 
7
  In addition, we provide the three models on two architectures small and base. For all models, we use a learning rate of 0.01, a batch size of 128 sequences, and a maximum sequence length of 512 whereas AraT5-tweet 128 maximum sequence is used. Hence, the original implementation of T5 in the TensorFlow framework is used to train the models. We train the models for 1M steps.8 Training took ∼ 80 days on 1 on Google Cloud TPU with 8 cores (v3.8) from TensorFlow Research Cloud (TFRC).