Update README.md
Browse filesAdd Optimum + ONNX usage and English corrections
README.md
CHANGED
@@ -14,19 +14,21 @@ widget:
|
|
14 |
DistilCamemBERT-QA
|
15 |
==================
|
16 |
|
17 |
-
We present DistilCamemBERT-QA which is [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) fine-tuned for the Question-Answering task for the french language. This model is
|
18 |
|
19 |
-
This modelization is close to [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) based on [CamemBERT](https://huggingface.co/camembert-base) model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase for example. Indeed, inference cost can be a technological issue especially
|
20 |
|
21 |
Dataset
|
22 |
-------
|
23 |
|
24 |
-
The dataset
|
25 |
|
26 |
Evaluation results and benchmark
|
27 |
--------------------------------
|
28 |
|
29 |
-
We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-base-qa) to two other modelizations working on french language. The first one [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) is based on well named [CamemBERT](https://huggingface.co/camembert-base), the french RoBERTa model and the second one [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) is based on [FlauBERT](https://huggingface.co/flaubert/flaubert_base_uncased)
|
|
|
|
|
30 |
|
31 |
| **model** | **time (ms)** | **exact match (%)** | **f1-score (%)** | **inclusion-score (%)** |
|
32 |
| :--------------: | :-----------: | :--------------: | :------------: | :------------: |
|
@@ -34,7 +36,7 @@ We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-b
|
|
34 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 432.17 | **59.76** | **79.57** | **69.23** |
|
35 |
| [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) | 875.84 | 0.22 | 5.21 | 3.68 |
|
36 |
|
37 |
-
Do not take into account the results
|
38 |
|
39 |
How to use DistilCamemBERT-QA
|
40 |
------------------------------
|
@@ -71,6 +73,20 @@ result
|
|
71 |
'answer': ' réalisateur et producteur américain.'}
|
72 |
```
|
73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
74 |
Citation
|
75 |
--------
|
76 |
```bibtex
|
|
|
14 |
DistilCamemBERT-QA
|
15 |
==================
|
16 |
|
17 |
+
We present DistilCamemBERT-QA, which is [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) fine-tuned for the Question-Answering task for the french language. This model is built using two datasets, FQuAD v1.0 and Piaf, composed of contexts and questions with their answers inside the context.
|
18 |
|
19 |
+
This modelization is close to [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) based on [CamemBERT](https://huggingface.co/camembert-base) model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase, for example. Indeed, inference cost can be a technological issue, especially in cross-encoding like this task. To counteract this effect, we propose this modelization which divides the inference time by 2 with the same consumption power, thanks to DistilCamemBERT.
|
20 |
|
21 |
Dataset
|
22 |
-------
|
23 |
|
24 |
+
The dataset comprises FQuAD v1.0 and Piaf with 24'566 questions and answers for the training set and 3'188 for the evaluation set.
|
25 |
|
26 |
Evaluation results and benchmark
|
27 |
--------------------------------
|
28 |
|
29 |
+
We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-base-qa) to two other modelizations working on the french language. The first one [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) is based on well named [CamemBERT](https://huggingface.co/camembert-base), the french RoBERTa model and the second one [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) is based on [FlauBERT](https://huggingface.co/flaubert/flaubert_base_uncased) another french model based on BERT architecture this time.
|
30 |
+
|
31 |
+
For our benchmarks, we compare matching character by character between the predicted answer and the ground truth. We also use the f1-score, which measures the intersection quality between predicted responses and ground truth. Finally, we use the inclusion score, which measures if the ground truth answer is included in the predicted answer. An **AMD Ryzen 5 4500U @ 2.3GHz with 6 cores** was used for the mean inference time measure.
|
32 |
|
33 |
| **model** | **time (ms)** | **exact match (%)** | **f1-score (%)** | **inclusion-score (%)** |
|
34 |
| :--------------: | :-----------: | :--------------: | :------------: | :------------: |
|
|
|
36 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 432.17 | **59.76** | **79.57** | **69.23** |
|
37 |
| [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) | 875.84 | 0.22 | 5.21 | 3.68 |
|
38 |
|
39 |
+
Do not take into account the results of the FlauBERT model. The modeling seems to be a problem, as the results seem very low.
|
40 |
|
41 |
How to use DistilCamemBERT-QA
|
42 |
------------------------------
|
|
|
73 |
'answer': ' réalisateur et producteur américain.'}
|
74 |
```
|
75 |
|
76 |
+
### Optimum + ONNX
|
77 |
+
|
78 |
+
```python
|
79 |
+
from optimum.onnxruntime import ORTModelForQuestionAnswering
|
80 |
+
from transformers import AutoTokenizer, pipeline
|
81 |
+
|
82 |
+
tokenizer = AutoTokenizer.from_pretrained("cmarkea/distilcamembert-base-qa")
|
83 |
+
model = ORTModelForQuestionAnswering.from_pretrained("cmarkea/distilcamembert-base-qa")
|
84 |
+
onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
|
85 |
+
|
86 |
+
# Quantized onnx model
|
87 |
+
quantized_model = ORTModelForQuestionAnswering.from_pretrained("cmarkea/distilcamembert-base-qa", file_name="model_quantized.onnx")
|
88 |
+
```
|
89 |
+
|
90 |
Citation
|
91 |
--------
|
92 |
```bibtex
|