|
--- |
|
license: cc-by-sa-4.0 |
|
language: |
|
- ja |
|
--- |
|
# Model card for model ID |
|
|
|
This is a T5 v1.1 model, pre-trained on a Japanese corpus. |
|
|
|
## Model details |
|
|
|
T5 is a Transformer-based Encoder-Decoder model, now in v1.1, with the following improvements over the original T5. |
|
- GEGLU activation in feed-forward hidden layer, rather than ReLU - see https://arxiv.org/abs/2002.05202 . |
|
- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning. |
|
- no parameter sharing between embedding and classifier layer |
|
- "xl" and "xxl" replace "3B" and "11B". The model shapes are a bit different - larger d_model and smaller num_heads and d_ff. |
|
|
|
This model is based on T5 v1.1. It was pre-trained on a Japanese corpus. For the Japanese corpus, Japanese Wikipedia and mC4/ja were used. |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** Retrieva, Inc. |
|
- **Model type:** T5 v1.1 |
|
- **Language(s) (NLP):** Japanese |
|
- **License:** CC-BY-SA 4.0 |
|
|
|
|
|
## Training Details |
|
|
|
We use T5X (https://github.com/google-research/t5x) for the training of this model, and it has been converted to the Huggingface transformer format. |
|
|
|
## Training Data |
|
|
|
The training data used is |
|
- The Japanese part of the multilingual C4(mC4/ja). |
|
- Japanese Wikipedia(20220920). |
|
|
|
#### Preprocessing |
|
The following filtering is done |
|
- Remove documents that do not use a single hiragana character. This removes English-only documents and documents in Chinese. |
|
- Whitelist-style filtering using TLD of URL to remove affiliate sites. |
|
|
|
#### Training Hyperparameters |
|
|
|
- dropout rate: 0.0 |
|
- batch size: 256 |
|
- fp32 |
|
- input length: 512 |
|
- output length: 114 |
|
|
|
- Otherwise, the default value of T5X (https://github.com/google-research/t5x/blob/main/t5x/examples/t5/t5_1_1/small.gin) is followed, including the following. |
|
- optimizer: Adafactor |
|
- base_learning_rate: 1.0 |
|
- warmup steps: 10000 |
|
|
|
#### Speeds, Sizes, Times |
|
|
|
We trained 1048576 steps. |
|
|
|
## Technical Specifications |
|
|
|
### Model Architecture and Objective |
|
Model architecture. |
|
- T5 v1.1(https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) |
|
- Size: Small(~77 million parameters) |
|
|
|
### Compute Infrastructure |
|
|
|
Google Cloud TPU v4-8. |
|
|
|
#### Software |
|
|
|
- T5X(https://github.com/google-research/t5x). |
|
|
|
## More Information |
|
|
|
https://note.com/retrieva/n/n7b4186dc5ada (in Japanese) |
|
|
|
## Model Card Authors |
|
|
|
Jiro Nishitoba |
|
|
|
## Model Card Contact |
|
|
|
pr@retrieva.jp |
|
|
