SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 4 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
product policy	'If I receive a defective Choker, what is the process to get a replacement?' 'Are there any restocking fees for returning a Choker?' 'What warranty do you offer on Choker products?'
product faq	'What sizes is the Sheer Heart Ring available in, and can you provide the price for each size?' 'Is the Silver Eye Pendant nickel-free and hypoallergenic?' 'What material is used for the Crystal Drop Earring, and how should I take care of it to prevent tarnishing?'
order tracking	"I haven't received an update on my order status for the Rosé Bloom Ring. Could you please provide me with the tracking details?" "I recently ordered the Pakhi Handcrafted Earring but I haven't received any shipping confirmation. Could you please update me on the status of my order?" "I recently ordered a Whispering Star Silver Ring, but I haven't received any shipment updates. Can you please provide me with the status of my order?"
product discoveribility	'What are the latest trends in bracelets that you have in stock?' "I'm interested in pendant sets from your 'Gold Plated Jewellery' collection. What options do you offer?" "I'm interested in silver bracelets. What options are available in that material?"

Evaluation

Metrics

Label	Accuracy
all	0.8025

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("What are the latest trends in bracelets that you have in stock?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	8	16.8438	31

Label	Training Sample Count
order tracking	8
product discoveribility	8
product faq	8
product policy	8

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (4, 4)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0208	1	0.1273	-
1.0417	50	0.004	-
2.0833	100	0.0005	-
3.125	150	0.0005	-

Framework Versions

Python: 3.9.16
SetFit: 1.0.3
Sentence Transformers: 2.7.0
Transformers: 4.40.1
PyTorch: 2.3.0
Datasets: 2.19.0
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Shankhdhar
/

classifier_test_model