|
--- |
|
language: en |
|
license: mit |
|
datasets: |
|
- conll2003 |
|
widget: |
|
- text: My name is Clara Clever and I live in Berkeley , California . |
|
--- |
|
|
|
# T5 Base Model for Named Entity Recognition (NER, CoNLL-2003) |
|
|
|
In this repository, we open source a T5 Base model, that was fine-tuned on the official CoNLL-2003 NER dataset. |
|
|
|
We use the great [TANL library](https://github.com/amazon-research/tanl) from Amazon for fine-tuning the model. |
|
|
|
The exact approach of fine-tuning is presented in the "TANL: Structured Prediction as Translation between Augmented Natural Languages" |
|
paper from Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang and Stefano Soatto. |
|
|
|
# Fine-Tuning |
|
|
|
We use the same hyper-parameter settings as used in the official implementation with one minor change. Instead of using 8 V100 GPUs, we train the model |
|
on one V100 GPU and used gradient accumulation. The slighly modified configuration file (`config.ini`) then looks like: |
|
|
|
```ini |
|
[conll03] |
|
datasets = conll03 |
|
model_name_or_path = t5-base |
|
num_train_epochs = 10 |
|
max_seq_length = 256 |
|
max_seq_length_eval = 512 |
|
per_device_train_batch_size = 4 |
|
per_device_eval_batch_size = 4 |
|
do_train = True |
|
do_eval = True |
|
do_predict = True |
|
gradient_accumulation_steps = 8 |
|
``` |
|
|
|
It took around 2 hours to fine-tune that model on the 14,041 training sentences of CoNLL-2003 dataset. |
|
|
|
# Evaluation |
|
|
|
On the development set, the following evaluation results could be achieved: |
|
|
|
```json |
|
{ |
|
"entity_precision": 0.9536446086664427, |
|
"entity_recall": 0.9555705149781218, |
|
"entity_f1": 0.9546065904505716, |
|
"entity_precision_no_type": 0.9773261672824992, |
|
"entity_recall_no_type": 0.9792998990238977, |
|
"entity_f1_no_type": 0.9783120376597176 |
|
} |
|
``` |
|
|
|
The evaluation results on the test set looks like: |
|
|
|
```json |
|
{ |
|
"entity_precision": 0.912182296231376, |
|
"entity_recall": 0.9213881019830028, |
|
"entity_f1": 0.9167620893155995, |
|
"entity_precision_no_type": 0.953900087642419, |
|
"entity_recall_no_type": 0.9635269121813032, |
|
"entity_f1_no_type": 0.9586893332158901 |
|
} |
|
``` |
|
|
|
To summarize: On the development set, 95.46% F1-Score and 91.68% on test set were achieved with this model. The paper reported a F1-Score of 91.7%. |
|
|
|
# License |
|
|
|
The models is licensed under [MIT](https://choosealicense.com/licenses/mit/). |
|
|
|
# Acknowledgments |
|
|
|
Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team, |
|
it is possible to download both cased and uncased models from their S3 storage 🤗 |
|
|
