File size: 14,781 Bytes
effc10e 4e5f4b6 effc10e 51f300d c39ea9d 4e5f4b6 1f9b5e3 a974e8f 06626f3 a974e8f 27979bb 6f5cf97 8122c96 27979bb 8d529a8 27979bb 8122c96 a81048c 8122c96 0c167b2 8122c96 0c167b2 8122c96 0c167b2 27979bb 8122c96 27979bb 5aa957c e2c9779 5aa957c 4995a9a 5aa957c 4995a9a 5aa957c 4995a9a 5aa957c 4995a9a 5aa957c 4995a9a 5aa957c 4995a9a e2c9779 4995a9a e2c9779 4995a9a e2c9779 4995a9a e2c9779 4995a9a e2c9779 4995a9a e2c9779 4995a9a e2c9779 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 |
---
language:
- en
license: apache-2.0
base_model:
- FacebookAI/roberta-base
pipeline_tag: token-classification
library_name: transformers
---
# Training
This model is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms.
The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed.
## Datasets
This model has been trained on the following datasets:
1. Aspect Based Sentiment Analysis SemEval Shared Tasks ([2014](https://aclanthology.org/S14-2004/), [2015](https://aclanthology.org/S15-2082/), [2016](https://aclanthology.org/S16-1002/))
2. Multi-Aspect Multi-Sentiment [MAMS](https://aclanthology.org/D19-1654/)
# Use
* Using the pipeline directly for end-to-end inference:
```python
from transformers import pipeline
ate_sent_pipeline = pipeline(task='ner',
aggregation_strategy='simple',
model="gauneg/roberta-base-absa-ate-sentiment")
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
ate_sent_pipeline(text_input)
```
* pipeline output:
```bash
[{'entity_group': 'pos', #sentiment polarity
'score': 0.8447307,
'word': ' food', # aspect term
'start': 26,
'end': 30},
{'entity_group': 'neg', #sentiment polarity
'score': 0.81927896,
'word': ' service', #aspect term
'start': 56,
'end': 63}]
```
# OR
* Making token level inferences with Auto classes
```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
model_id = "gauneg/roberta-base-absa-ate-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)
# the sequence of labels used during training
labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
id2lab = {idx: lab for lab, idx in labels.items()}
lab2id = {lab: idx for lab, idx in labels.items()}
model = AutoModelForTokenClassification.from_pretrained(model_id,
num_labels=len(labels), id2label=id2lab, label2id=lab2id)
# making one prediction at a time (should be padded/batched and truncated for efficiency)
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
tok_inputs = tokenizer(text_input, return_tensors="pt")
y_pred = model(**tok_inputs) # predicting the logits
# since first and the last tokens are excluded (<s> and </s>)
# they have to be removed before decoding the labels predicted against them
y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits
decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin]
## displaying the input tokens with predictions and skipping <s> and </s> tokens at the beginning and the end respectively
decoded_toks = tok_inputs['input_ids'][0][1:-1]
tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))
```
* results in `tok_level_pred` variable
```bash
[('Be', 'O'),
('en', 'O'),
('Ġhere', 'O'),
('Ġa', 'O'),
('Ġfew', 'O'),
('Ġtimes', 'O'),
('Ġand', 'O'),
('Ġfood', 'B-pos'),
('Ġhas', 'O'),
('Ġalways', 'O'),
('Ġbeen', 'O'),
('Ġgood', 'O'),
('Ġbut', 'O'),
('Ġservice', 'B-neg'),
('Ġreally', 'O'),
('Ġsuffers', 'O'),
('Ġwhen', 'O'),
('Ġit', 'O'),
('Ġgets', 'O'),
('Ġcrowded', 'O'),
('.', 'O')]
```
# Evaluation on Benchmark Test Datasets
The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)
# ATE (Aspect Term Extraction Only)
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35|
|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75|
|MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79|
# Aspect Sentiment Evaluation
This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49|
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26|
|MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25|
|