metadata

library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
datasets:
  - Ramyashree/Dataset-train500-test100withwronginput
metrics:
  - accuracy
widget:
  - text: I weant to use my other  account, switch them
  - text: I can't remember my password, help me reset it
  - text: the game was postponed and i wanna get a reimbursement
  - text: where to change to another online account
  - text: the show was cancelled, get a reimbursement
pipeline_tag: text-classification
inference: true
base_model: thenlper/gte-large
model-index:
  - name: SetFit with thenlper/gte-large
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Ramyashree/Dataset-train500-test100withwronginput
          type: Ramyashree/Dataset-train500-test100withwronginput
          split: test
        metrics:
          - type: accuracy
            value: 0.94
            name: Accuracy

SetFit with thenlper/gte-large

This is a SetFit model trained on the Ramyashree/Dataset-train500-test100withwronginput dataset that can be used for Text Classification. This SetFit model uses thenlper/gte-large as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: thenlper/gte-large
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 10 classes
Training Dataset: Ramyashree/Dataset-train500-test100withwronginput

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
create_account	"I don't have an online account, what do I have to do to register?" 'can you tell me if i can regisger two accounts with a single email address?' 'I have no online account, open one, please'
edit_account	'how can I modify the information on my profile?' 'can u ask an agent how to make changes to my profile?' 'I want to update the information on my profile'
delete_account	'can I close my account?' "I don't want my account, can you delete it?" 'how do i close my online account?'
switch_account	'I would like to use my other online account , could you switch them, please?' 'i want to use my other online account, can u change them?' 'how do i change to another account?'
get_invoice	'what can you tell me about getting some bills?' 'tell me where I can request a bill' 'ask an agent if i can obtain some bills'
get_refund	'the game was postponed, help me obtain a reimbursement' 'the game was postponed, what should I do to obtain a reimbursement?' 'the concert was postponed, what should I do to request a reimbursement?'
payment_issue	'i have an issue making a payment with card and i want to inform of it, please' 'I got an error message when I attempted to pay, but my card was charged anyway and I want to notify it' 'I want to notify a problem making a payment, can you help me?'
check_refund_policy	"I'm interested in your reimbursement polivy" 'i wanna see your refund policy, can u help me?' 'where do I see your money back policy?'
recover_password	'my online account was hacked and I want tyo get it back' "I lost my password and I'd like to retrieve it, please" 'could u ask an agent how i can reset my password?'
track_refund	'tell me if my refund was processed' 'I need help checking the status of my refund' 'I want to see the status of my refund, can you help me?'

Evaluation

Metrics

Label	Accuracy
all	0.94

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Ramyashree/gte-large-train-test-3")
# Run inference
preds = model("where to change to another online account")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	10.258	24

Label	Training Sample Count
check_refund_policy	50
create_account	50
delete_account	50
edit_account	50
get_invoice	50
get_refund	50
payment_issue	50
recover_password	50
switch_account	50
track_refund	50

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0008	1	0.3248	-
0.04	50	0.1606	-
0.08	100	0.0058	-
0.12	150	0.0047	-
0.16	200	0.0009	-
0.2	250	0.0007	-
0.24	300	0.001	-
0.28	350	0.0008	-
0.32	400	0.0005	-
0.36	450	0.0004	-
0.4	500	0.0005	-
0.44	550	0.0005	-
0.48	600	0.0006	-
0.52	650	0.0005	-
0.56	700	0.0004	-
0.6	750	0.0004	-
0.64	800	0.0002	-
0.68	850	0.0003	-
0.72	900	0.0002	-
0.76	950	0.0002	-
0.8	1000	0.0003	-
0.84	1050	0.0002	-
0.88	1100	0.0002	-
0.92	1150	0.0003	-
0.96	1200	0.0003	-
1.0	1250	0.0003	-

Framework Versions

Python: 3.10.12
SetFit: 1.0.1
Sentence Transformers: 2.2.2
Transformers: 4.35.2
PyTorch: 2.1.0+cu121
Datasets: 2.15.0
Tokenizers: 0.15.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}