julien-c HF staff commited on
Commit
04d7163
1 Parent(s): 6570e85

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/mrm8488/spanbert-finetuned-squadv2/README.md

Files changed (1) hide show
  1. README.md +86 -0
README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ thumbnail:
4
+ ---
5
+
6
+ # SpanBERT (spanbert-base-cased) fine-tuned on SQuAD v2
7
+
8
+ [SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) for **Q&A** downstream task.
9
+
10
+ ## Details of SpanBERT
11
+
12
+ [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
13
+
14
+ ## Details of the downstream task (Q&A) - Dataset
15
+
16
+ [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
17
+
18
+ | Dataset | Split | # samples |
19
+ | -------- | ----- | --------- |
20
+ | SQuAD2.0 | train | 130k |
21
+ | SQuAD2.0 | eval | 12.3k |
22
+
23
+ ## Model training
24
+
25
+ The model was trained on a Tesla P100 GPU and 25GB of RAM.
26
+ The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)
27
+
28
+ ## Results:
29
+
30
+ | Metric | # Value |
31
+ | ------ | --------- |
32
+ | **EM** | **78.80** |
33
+ | **F1** | **82.22** |
34
+
35
+ ### Raw metrics:
36
+
37
+ ```json
38
+ {
39
+ "exact": 78.80064010780762,
40
+ "f1": 82.22801347271162,
41
+ "total": 11873,
42
+ "HasAns_exact": 78.74493927125506,
43
+ "HasAns_f1": 85.60951483831069,
44
+ "HasAns_total": 5928,
45
+ "NoAns_exact": 78.85618166526493,
46
+ "NoAns_f1": 78.85618166526493,
47
+ "NoAns_total": 5945,
48
+ "best_exact": 78.80064010780762,
49
+ "best_exact_thresh": 0.0,
50
+ "best_f1": 82.2280134727116,
51
+ "best_f1_thresh": 0.0
52
+ }
53
+ ```
54
+
55
+ ## Comparison:
56
+
57
+ | Model | EM | F1 score |
58
+ | ----------------------------------------------------------------------------------------- | --------- | --------- |
59
+ | [SpanBert official repo](https://github.com/facebookresearch/SpanBERT#pre-trained-models) | - | 83.6\* |
60
+ | [spanbert-finetuned-squadv2](https://huggingface.co/mrm8488/spanbert-finetuned-squadv2) | **78.80** | **82.22** |
61
+
62
+ ## Model in action
63
+
64
+ Fast usage with **pipelines**:
65
+
66
+ ```python
67
+ from transformers import pipeline
68
+
69
+ qa_pipeline = pipeline(
70
+ "question-answering",
71
+ model="mrm8488/spanbert-finetuned-squadv2",
72
+ tokenizer="mrm8488/spanbert-finetuned-squadv2"
73
+ )
74
+
75
+ qa_pipeline({
76
+ 'context': "Manuel Romero has been working hardly in the repository hugginface/transformers lately",
77
+ 'question': "Who has been working hard for hugginface/transformers lately?"
78
+
79
+ })
80
+
81
+ # Output: {'answer': 'Manuel Romero','end': 13,'score': 6.836378586818937e-09, 'start': 0}
82
+ ```
83
+
84
+ > Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
85
+
86
+ > Made with <span style="color: #e25555;">&hearts;</span> in Spain