IProject-10/deberta-base-finetuned-squad2

Model description

DeBERTabase fine-tuned on SQuAD 2.0 : Encoder-based Transformer Language model. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.
Suitable for Question-Answering tasks, predicts answer spans within the context provided.

Language model: microsoft/deberta-base
Language: English
Downstream-task: Question-Answering
Training data: Train-set SQuAD 2.0
Evaluation data: Evaluation-set SQuAD 2.0
Hardware Accelerator used: GPU Tesla T4

Intended uses & limitations

For Question-Answering -

!pip install transformers
from transformers import pipeline
model_checkpoint = "IProject-10/deberta-base-finetuned-squad2"
question_answerer = pipeline("question-answering", model=model_checkpoint)

context = """
🤗 Transformers is backed by the three most popular deep learning libraries — Jax, PyTorch and TensorFlow — with a seamless integration
between them. It's straightforward to train your models with one before loading them for inference with the other.
"""

question = "Which deep learning libraries back 🤗 Transformers?"
question_answerer(question=question, context=context)

Results

Evaluation on SQuAD 2.0 validation dataset:

 exact: 81.03259496336226,
 f1: 84.42279239924598,
 total: 11873,
 HasAns_exact: 79.30161943319838,
 HasAns_f1: 86.09173653108105,
 HasAns_total: 5928,
 NoAns_exact: 82.75862068965517,
 NoAns_f1: 82.75862068965517,
 NoAns_total: 5945,
 best_exact: 81.03259496336226,
 best_exact_thresh: 0.9992604851722717,
 best_f1: 84.42279239924635,
 best_f1_thresh: 0.9992604851722717,
 total_time_in_seconds: 326.41847440000004,
 samples_per_second: 36.37355398411236,
 latency_in_seconds: 0.027492501844521185

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.8054	1.0	8238	0.7902
0.5368	2.0	16476	0.7901
0.3845	3.0	24714	0.9334

This model is a fine-tuned version of microsoft/deberta-base on the squad_v2 dataset. It achieves the following results on the evaluation set:

• Loss: 0.9334

Framework versions

• Transformers 4.31.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.3
• Tokenizers 0.13.3