|
--- |
|
license: apache-2.0 |
|
language: |
|
- ko |
|
--- |
|
|
|
# Korean ALBERT |
|
|
|
# Dataset |
|
- [AI-HUB](https://www.aihub.or.kr/) |
|
- [국립국어원 - 모두의 말뭉치](https://kli.korean.go.kr/corpus/main/requestMain.do?lang=ko) |
|
- [Korean News Comments](https://www.kaggle.com/junbumlee/kcbert-pretraining-corpus-korean-news-comments) |
|
|
|
|
|
# Evaluation results |
|
- The code for finetuning can be found at [KcBERT-Finetune](https://github.com/Beomi/KcBERT-finetune). |
|
|
|
| | Size(용량) | Average Score | **NSMC**<br/>(acc) | **Naver NER**<br/>(F1) | **PAWS**<br/>(acc) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | **Question Pair**<br/>(acc) | **KorQuaD (Dev)**<br/>(EM/F1) | |
|
|:---------------------- |:----------:|:-------------:|:------------------:|:----------------------:|:------------------:|:--------------------:|:-------------------------:|:---------------------------:|:-----------------------------:| |
|
| KcELECTRA-base | 475M | 84.84 | 91.71 | 86.90 | 74.80 | 81.65 | 82.65 | **95.78** | 70.60 / 90.11 | |
|
| KcELECTRA-base-v2022 | 475M | 85.20 | **91.97** | **87.35** | 76.50 | **82.12** | **83.67** | 95.12 | 69.00 / 90.40 | |
|
| KcBERT-Base | 417M | 79.65 | 89.62 | 84.34 | 66.95 | 74.85 | 75.57 | 93.93 | 60.25 / 84.39 | |
|
| KcBERT-Large | 1.2G | 81.33 | 90.68 | 85.53 | 70.15 | 76.99 | 77.49 | 94.06 | 62.16 / 86.64 | |
|
| KoBERT | 351M | 82.21 | 89.63 | 86.11 | 80.65 | 79.00 | 79.64 | 93.93 | 52.81 / 80.27 | |
|
| XLM-Roberta-Base | 1.03G | 84.01 | 89.49 | 86.26 | 82.95 | 79.92 | 79.09 | 93.53 | 64.70 / 88.94 | |
|
| HanBERT | 614M | 86.24 | 90.16 | 87.31 | 82.40 | 80.89 | 83.33 | 94.19 | 78.74 / 92.02 | |
|
| KoELECTRA-Base | 423M | 84.66 | 90.21 | 86.87 | 81.90 | 80.85 | 83.21 | 94.20 | 61.10 / 89.59 | |
|
| KoELECTRA-Base-v2 | 423M | **86.96** | 89.70 | 87.02 | **83.90** | 80.61 | 84.30 | 94.72 | **84.34 / 92.58** | |
|
| DistilKoBERT | 108M | 76.76 | 88.41 | 84.13 | 62.55 | 70.55 | 73.21 | 92.48 | 54.12 / 77.80 | |
|
| **ko-albert-base-v1** | **51M** | 80.46 | 86.83 | 82.26 | 69.95 | 74.17 | 74.48 | 94.06 | 76.08 / 86.82 | |
|
| **ko-albert-large-v1** | **75M** | 82.39 | 86.91 | 83.12 | 76.10 | 76.01 | 77.46 | 94.33 | 77.64 / 87.99 | |
|
|
|
*The size of HanBERT is the sum of the BERT model and the tokenizer DB. |
|
|
|
*These results were obtained using the default configuration settings. Better performance may be achieved with additional hyperparameter tuning. |
|
|
|
|
|
# How to use |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
# Base Model (51M) |
|
tokenizer = AutoTokenizer.from_pretrained("lots-o/ko-albert-base-v1") |
|
model = AutoModel.from_pretrained("lots-o/ko-albert-base-v1") |
|
|
|
# Large Model (75M) |
|
tokenizer = AutoTokenizer.from_pretrained("lots-o/ko-albert-large-v1") |
|
model = AutoModel.from_pretrained("lots-o/ko-albert-large-v1") |
|
``` |
|
|
|
# Acknowledgement |
|
- The GCP/TPU environment used for training the ALBERT Model was supported by the [TRC](https://sites.research.google/trc/about/) program. |
|
|
|
# Reference |
|
## Paper |
|
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) |
|
|
|
## Github Repos |
|
- [google-albert](https://github.com/google-research/albert) |
|
- [albert-zh](https://github.com/brightmart/albert_zh) |
|
- [KcBERT](https://github.com/Beomi/KcBERT) |
|
- [KcBERT-Finetune](https://github.com/Beomi/KcBERT-finetune) |
