Hugging Face's logo

gauneg
/
roberta-large-absa-ate-sentiment-lora-adapter

Token Classification
Transformers
Safetensors
LoRA
Adapter
Inference Endpoints
Model card Files Community
roberta-large-absa-ate-sentiment-lora-adapter
File size: 15,360 Bytes 
747a559
 
 
 
 
 
c8bc735
 
 
724e84b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0791a75
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
724e84b
 
 
 
 
 
0791a75
724e84b
0791a75
 
724e84b
 
 
0791a75
 
 
 
 
 
724e84b
 
0791a75
 
 
 
724e84b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0791a75
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
724e84b
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
---
license: mit
base_model:
- FacebookAI/roberta-large
pipeline_tag: token-classification
library_name: transformers
tags:
- LoRA
- Adapter
---

# Training 
This model adapter is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms. 
The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed. 
It has been created using [PEFT](https://huggingface.co/docs/peft/index) framework for [LoRA:Low-Rank Adaptation](https://arxiv.org/abs/2106.09685).

## Datasets
This model has been trained on the following datasets:

1. Aspect Based Sentiment Analysis SemEval Shared Tasks ([2014](https://aclanthology.org/S14-2004/), [2015](https://aclanthology.org/S15-2082/), [2016](https://aclanthology.org/S16-1002/))
2. Multi-Aspect Multi-Sentiment [MAMS](https://aclanthology.org/D19-1654/)

# Use

* Loading the base model and merging it with LoRA parameters

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from peft import PeftModel

# preparing the labels
labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
id2labels = {k:lab for lab, k in labels.items()}
labels2ids = {k:lab for lab, k in id2labels.items()}

# loading tokenizer and base_model
base_id = 'FacebookAI/roberta-large'
tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForTokenClassification.from_pretrained(base_id, num_labels=len(labels), id2label=id2labels, label2id=labels2ids)

# using this adapter with base model 
model = PeftModel.from_pretrained(base_model, 'gauneg/roberta-large-absa-ate-sentiment-lora-adapter', is_trainable=False)

```

This model can be utilized in the following two methods:

1. Using pipelines for end to end inference
2. Making token level inference


## Using end-to-end token classification pipeline

```python
# after loading base model and the adapter as shown in the previous snippet
from transformers import pipeline

ate_senti_pipeline = pipeline(task='ner', 
                  aggregation_strategy='simple',
                  model=model,
                  tokenizer=tokenizer)


text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
ate_senti_pipeline(text_input)

```
OUTPUT
```bash
[{'entity_group': 'pos',
  'score': 0.92310727,
  'word': ' food',
  'start': 26,
  'end': 30},
 {'entity_group': 'neg',
  'score': 0.90695626,
  'word': ' service',
  'start': 56,
  'end': 63}]
```

# OR

## Making token level inference

```python
# after loading base model and the adapter as shown in the previous snippet

# making one prediction at a time (should be padded/batched and truncated for efficiency)
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
tok_inputs = tokenizer(text_input, return_tensors="pt")


y_pred = model(**tok_inputs) # predicting the logits

# since first and the last tokens are excluded (<s> and </s>) labels predicted against them
# should be removed before decoding predicted labels
y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits


decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin]


## displaying the input tokens with predictions and skipping <s> and </s> tokens at the beginning and the end respectively
decoded_toks = tok_inputs['input_ids'][0][1:-1]
tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))

```

RESULTS in `tok_levl_pred` variable:
```bash
[('Be', 'O'),
 ('en', 'O'),
 ('Ġhere', 'O'),
 ('Ġa', 'O'),
 ('Ġfew', 'O'),
 ('Ġtimes', 'O'),
 ('Ġand', 'O'),
 ('Ġfood', 'B-pos'),
 ('Ġhas', 'O'),
 ('Ġalways', 'O'),
 ('Ġbeen', 'O'),
 ('Ġgood', 'O'),
 ('Ġbut', 'O'),
 ('Ġservice', 'B-neg'),
 ('Ġreally', 'O'),
 ('Ġsuffers', 'O'),
 ('Ġwhen', 'O'),
 ('Ġit', 'O'),
 ('Ġgets', 'O'),
 ('Ġcrowded', 'O'),
 ('.', 'O')]
```



# Evaluation on Benchmark Test Datasets

The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)

# ATE (Aspect Term Extraction Only)
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35|
|MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75|
|MAMS-ATE (2019)|FacebookAI/roberta-large|(this)[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79|

# Aspect Sentiment Evaluation
This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)
| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08|
|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03|
|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81|
|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93|
|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49|
|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26|
|MAMS-ATE (2019)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07|
|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74|
|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93|
|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11|
|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29|
|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12|
| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05|
|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15|
|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25|