|
--- |
|
tags: |
|
- exbert |
|
license: apache-2.0 |
|
--- |
|
|
|
# ouBioBERT-Base, Uncased |
|
|
|
Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) is a language model based on the BERT-Base (Devlin, et al., 2019) architecture. We pre-trained ouBioBERT on PubMed abstracts from the PubMed baseline (ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline) via our method. |
|
|
|
The details of the pre-training procedure can be found in Wada, et al. (2020). |
|
|
|
## Evaluation |
|
|
|
We evaluated the performance of ouBioBERT in terms of the biomedical language understanding evaluation (BLUE) benchmark (Peng, et al., 2019). The numbers are mean (standard deviation) on five different random seeds. |
|
|
|
|
|
| Dataset | Task Type | Score | |
|
|:----------------|:-----------------------------|-------------:| |
|
| MedSTS | Sentence similarity | 84.9 (0.6) | |
|
| BIOSSES | Sentence similarity | 92.3 (0.8) | |
|
| BC5CDR-disease | Named-entity recognition | 87.4 (0.1) | |
|
| BC5CDR-chemical | Named-entity recognition | 93.7 (0.2) | |
|
| ShARe/CLEFE | Named-entity recognition | 80.1 (0.4) | |
|
| DDI | Relation extraction | 81.1 (1.5) | |
|
| ChemProt | Relation extraction | 75.0 (0.3) | |
|
| i2b2 2010 | Relation extraction | 74.0 (0.8) | |
|
| HoC | Document classification | 86.4 (0.5) | |
|
| MedNLI | Inference | 83.6 (0.7) | |
|
| **Total** | Macro average of the scores |**83.8 (0.3)**| |
|
|
|
|
|
## Code for Fine-tuning |
|
We made the source code for fine-tuning freely available at [our repository](https://github.com/sy-wada/blue_benchmark_with_transformers). |
|
|
|
## Citation |
|
|
|
If you use our work in your research, please kindly cite the following paper: |
|
|
|
```bibtex |
|
@misc{2005.07202, |
|
Author = {Shoya Wada and Toshihiro Takeda and Shiro Manabe and Shozo Konishi and Jun Kamohara and Yasushi Matsumura}, |
|
Title = {A pre-training technique to localize medical BERT and enhance BioBERT}, |
|
Year = {2020}, |
|
Eprint = {arXiv:2005.07202}, |
|
} |
|
``` |
|
|
|
<a href="https://huggingface.co/exbert/?model=seiya/oubiobert-base-uncased&sentence=Coronavirus%20disease%20(COVID-19)%20is%20caused%20by%20SARS-COV2%20and%20represents%20the%20causative%20agent%20of%20a%20potentially%20fatal%20disease%20that%20is%20of%20great%20global%20public%20health%20concern."> |
|
<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png"> |
|
</a> |
|
|
