SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 3 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
neutral	'ordered my new shirt' 'Yay got the Internet on my itouch working' 'Getting ready for work and the sun is shining plus its the w e Bgt tonight what am I gon na do after its finished '
positive	'Finally home after a night of dinner and drinking with friends Going to sleep now hoping the bed doesnt spin too much ' ' Thank you I love my tattoos they are all very special to me My feet ones are beautiful ' 'Day is going well so far Meeting until four though '
negative	' Oh final msg Why didnt you review my boardgame BookchaseA AA12 when you were on telly We didnt even get a nice letter ' 'have to wear my glasses today cos my right eye is swollen and i dont know why' ' how crappy for him'

Evaluation

Metrics

Label	Accuracy
all	0.7301

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("subham18/setfit-paraphrase-mpnet-base-v2-twitter-sentiment-cleaned-73")
# Run inference
preds = model("I still miss him And i do nt think hes coming back")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	2	13.9	31

Label	Training Sample Count
Negative	0
Positive	0
Neutral	0

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (4, 4)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0011	1	0.3222	-
0.0533	50	0.223	-
0.1066	100	0.2817	-
0.1599	150	0.1102	-
0.2132	200	0.1271	-
0.2665	250	0.0307	-
0.3198	300	0.0013	-
0.3731	350	0.0006	-
0.4264	400	0.0006	-
0.4797	450	0.0004	-
0.5330	500	0.0006	-
0.5864	550	0.0002	-
0.6397	600	0.0003	-
0.6930	650	0.0002	-
0.7463	700	0.0002	-
0.7996	750	0.0002	-
0.8529	800	0.0002	-
0.9062	850	0.0002	-
0.9595	900	0.0005	-
1.0	938	-	0.2816
1.0128	950	0.0001	-
1.0661	1000	0.0027	-
1.1194	1050	0.0002	-
1.1727	1100	0.0002	-
1.2260	1150	0.0001	-
1.2793	1200	0.0003	-
1.3326	1250	0.0001	-
1.3859	1300	0.0002	-
1.4392	1350	0.0001	-
1.4925	1400	0.0001	-
1.5458	1450	0.0001	-
1.5991	1500	0.0001	-
1.6525	1550	0.0001	-
1.7058	1600	0.0001	-
1.7591	1650	0.0001	-
1.8124	1700	0.0001	-
1.8657	1750	0.0002	-
1.9190	1800	0.0001	-
1.9723	1850	0.0001	-
2.0	1876	-	0.2846
2.0256	1900	0.0001	-
2.0789	1950	0.0001	-
2.1322	2000	0.0001	-
2.1855	2050	0.0001	-
2.2388	2100	0.0001	-
2.2921	2150	0.0001	-
2.3454	2200	0.0002	-
2.3987	2250	0.0001	-
2.4520	2300	0.0001	-
2.5053	2350	0.0001	-
2.5586	2400	0.0001	-
2.6119	2450	0.0007	-
2.6652	2500	0.0001	-
2.7186	2550	0.0001	-
2.7719	2600	0.0002	-
2.8252	2650	0.0001	-
2.8785	2700	0.0001	-
2.9318	2750	0.0001	-
2.9851	2800	0.0001	-
3.0	2814	-	0.2843
3.0384	2850	0.0001	-
3.0917	2900	0.0001	-
3.1450	2950	0.0001	-
3.1983	3000	0.0001	-
3.2516	3050	0.0002	-
3.3049	3100	0.0001	-
3.3582	3150	0.0001	-
3.4115	3200	0.0001	-
3.4648	3250	0.0001	-
3.5181	3300	0.0	-
3.5714	3350	0.0001	-
3.6247	3400	0.0	-
3.6780	3450	0.0	-
3.7313	3500	0.0001	-
3.7846	3550	0.0001	-
3.8380	3600	0.0002	-
3.8913	3650	0.0001	-
3.9446	3700	0.0002	-
3.9979	3750	0.0	-
4.0	3752	-	0.2861

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.3
SetFit: 1.0.3
Sentence Transformers: 3.0.1
Transformers: 4.39.0
PyTorch: 2.4.0+cu121
Datasets: 2.21.0
Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

subham18
/

setfit-paraphrase-mpnet-base-v2-twitter-sentiment-cleaned-73