|
--- |
|
language: |
|
- en |
|
license: apache-2.0 |
|
base_model: |
|
- FacebookAI/roberta-base |
|
pipeline_tag: token-classification |
|
library_name: transformers |
|
--- |
|
|
|
# Training |
|
This model is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms. |
|
The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed. |
|
|
|
## Datasets |
|
This model has been trained on the following datasets: |
|
|
|
1. Aspect Based Sentiment Analysis SemEval Shared Tasks ([2014](https://aclanthology.org/S14-2004/), [2015](https://aclanthology.org/S15-2082/), [2016](https://aclanthology.org/S16-1002/)) |
|
2. Multi-Aspect Multi-Sentiment [MAMS](https://aclanthology.org/D19-1654/) |
|
|
|
# Use |
|
|
|
* Using the pipeline directly for end-to-end inference: |
|
```python |
|
from transformers import pipeline |
|
|
|
ate_sent_pipeline = pipeline(task='ner', |
|
aggregation_strategy='simple', |
|
model="gauneg/roberta-base-absa-ate-sentiment") |
|
|
|
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded." |
|
ate_sent_pipeline(text_input) |
|
``` |
|
* pipeline output: |
|
```bash |
|
[{'entity_group': 'pos', #sentiment polarity |
|
'score': 0.8447307, |
|
'word': ' food', # aspect term |
|
'start': 26, |
|
'end': 30}, |
|
{'entity_group': 'neg', #sentiment polarity |
|
'score': 0.81927896, |
|
'word': ' service', #aspect term |
|
'start': 56, |
|
'end': 63}] |
|
|
|
``` |
|
|
|
# OR |
|
* Making token level inferences with Auto classes |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForTokenClassification |
|
model_id = "gauneg/roberta-base-absa-ate-sentiment" |
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
|
|
# the sequence of labels used during training |
|
labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100} |
|
id2lab = {idx: lab for lab, idx in labels.items()} |
|
lab2id = {lab: idx for lab, idx in labels.items()} |
|
|
|
model = AutoModelForTokenClassification.from_pretrained(model_id, |
|
num_labels=len(labels), id2label=id2lab, label2id=lab2id) |
|
|
|
# making one prediction at a time (should be padded/batched and truncated for efficiency) |
|
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded." |
|
tok_inputs = tokenizer(text_input, return_tensors="pt") |
|
|
|
|
|
y_pred = model(**tok_inputs) # predicting the logits |
|
|
|
# since first and the last tokens are excluded (<s> and </s>) |
|
# they have to be removed before decoding the labels predicted against them |
|
y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits |
|
|
|
|
|
decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin] |
|
|
|
|
|
## displaying the input tokens with predictions and skipping <s> and </s> tokens at the beginning and the end respectively |
|
decoded_toks = tok_inputs['input_ids'][0][1:-1] |
|
tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred)) |
|
|
|
``` |
|
|
|
* results in `tok_level_pred` variable |
|
|
|
```bash |
|
[('Be', 'O'), |
|
('en', 'O'), |
|
('Ġhere', 'O'), |
|
('Ġa', 'O'), |
|
('Ġfew', 'O'), |
|
('Ġtimes', 'O'), |
|
('Ġand', 'O'), |
|
('Ġfood', 'B-pos'), |
|
('Ġhas', 'O'), |
|
('Ġalways', 'O'), |
|
('Ġbeen', 'O'), |
|
('Ġgood', 'O'), |
|
('Ġbut', 'O'), |
|
('Ġservice', 'B-neg'), |
|
('Ġreally', 'O'), |
|
('Ġsuffers', 'O'), |
|
('Ġwhen', 'O'), |
|
('Ġit', 'O'), |
|
('Ġgets', 'O'), |
|
('Ġcrowded', 'O'), |
|
('.', 'O')] |
|
``` |
|
|
|
|
|
|
|
# Evaluation on Benchmark Test Datasets |
|
|
|
The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens |
|
on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels) |
|
|
|
# ATE (Aspect Term Extraction Only) |
|
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score | |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6| |
|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07| |
|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0| |
|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21| |
|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31| |
|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08| |
|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35| |
|
|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39| |
|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75| |
|
|MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45| |
|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57| |
|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32| |
|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74| |
|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63| |
|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28| |
|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84| |
|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2| |
|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85| |
|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79| |
|
|
|
# Aspect Sentiment Evaluation |
|
This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens |
|
on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages) |
|
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score | |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94| |
|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08| |
|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03| |
|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81| |
|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09| |
|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93| |
|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49| |
|
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26| |
|
|MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07| |
|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74| |
|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13| |
|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93| |
|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11| |
|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6| |
|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29| |
|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12| |
|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- | |
|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05| |
|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33| |
|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15| |
|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25| |
|
|
|
|
|
|
