README.md

metadata

tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: I need to cancel an incorrect transaction.
  - text: The app reverted what I topped up.
  - text: Where can I use my card?
  - text: Why do I have to verify my identity?
  - text: Since my pin is blocked, would you help me unblock it?
metrics:
  - accuracy
  - f1_score
  - recall
  - precision
pipeline_tag: text-classification
library_name: setfit
inference: true
base_model: Alibaba-NLP/gte-multilingual-base
model-index:
  - name: SetFit with Alibaba-NLP/gte-multilingual-base
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.8964285714285715
            name: Accuracy
          - type: f1_score
            value: 0.8962133652169145
            name: F1_Score
          - type: recall
            value: 0.8964285714285715
            name: Recall
          - type: precision
            value: 0.9020279301598277
            name: Precision

SetFit with Alibaba-NLP/gte-multilingual-base

This is a SetFit model that can be used for Text Classification. This SetFit model uses Alibaba-NLP/gte-multilingual-base as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: Alibaba-NLP/gte-multilingual-base
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 8192 tokens
• Number of Classes: 77 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
33	'I might need to exchange between USD and GBP, is this possible with your app?' 'I may need to exchange my USD for GBP, can your app do that?' 'Can this app exchange American and English currency?'
15	'Why am I charged an extra fee on my card?' 'There was an extra charge when I used my card' 'Who charged me a fee for paying by card?'
52	'I would like to receive a refund for something I bought.' 'How do I return something for a full refund?' 'How do I begin the process of a refund?'
44	'I am unaware of what my password is.' 'Are there directions to get a new passcode if I forgot mine?' 'What are the steps for changing my passcode?'
73	'Can I have a mastercard?' 'Is it acceptable to use either Visa or Mastercard?' 'Visa and MasterCard, can I get them both?'
11	'How much longer is it going to take to get my new card?' "I don't have my card after 1 week. What are my next steps?" 'I need to track the card sent to me.'
22	'The card has suffered a security breach.' 'Someone besides me might be using my card.' "I have transactions that I don't recognize - I think someone is using my card."
72	"The virtual card won't work." 'I tried to use my disposable virtual card to pay a subscription to the gym and it got rejected. Any ideas why?' "I can't seem to get my virtual card to work."
41	'Someone took my card without permission' 'Is there any way for you to find my card?' 'I lost my wallet and all my cards were in it.'
59	"I added money but my top-up wasn't processed by the app." 'My top up has failed' 'I need assistance finding out why I am unable to top-up my card, it keeps getting refused. It is extremely imperative I get this resolved quickly.'
27	"Why do you keep declining my transfers? it's always been working really well so far but when I tried to buy something just now the card got declined, I tried couple of times already but same thing is happening." 'Why did my transfer not work, it says it was declined?' 'How can I check for an unsupported merchant for my transfer request?'
4	'How can I setup automatic top-up?' 'Is auto top up a provided service?' 'What are the restrictions on auto top-up?'
0	'I have a new card and need to activate it' 'I cannot activate my card.' 'Is my card ready for use or does it need activated and if so how?'
14	"I can't use my card because it is not working." "The card that I have isn't working. This is the physical card." 'My Netflix subscription says that there is an issue with my card information, but I thought everything was fine. Why would it be declined?'
23	"Are my contactless settings correct? I tried to use it today but it wasn't working." "Why wasn't my contactless accepted at the metro?" "Why wouldn't my contactless work for the metro?"
70	'Please explain to me how to know where my funds are from.' 'need to verify source of funds' 'where did my funds come from?'
40	'Are there virtual cards' 'I need my virtual card' 'Where would I order a virtual card?'
3	'Is there a Mastercard ATM nearby?' 'Where is the closest ATM to me?' 'Can this card be used at all ATMs?'
50	'How can my boss pay me directly to the card?' 'How can I receive money from my friend?' 'Is there any way to get my salary in a different currency?'
60	'Do I have a limit to the number of top-ups per day?' 'Do top-ups have limits?' 'How often can I top up?'
26	'The ATM cancelled a transaction on me when I was trying to get some cash out.' 'When going to the ATM to take out some cash today my withdrawal was declined. I need to fix this problem as soon as possible. whats going on?' "A withdraw I tried to make at the ATM was declined I don't know why."
28	"Help! In the app there is a direct debit that I don't remember." 'I just went through a old statement and noticed that a very large payment went through and i did not make it. It was from a few months ago, can I still dispute the payment?' "I see a direct debit that isn't mine? How can I get my money back?"
63	'Good day, in reviewing my credit card activity over the last several days, I see a repeated charge for a restaurant that I went to. The charge was put through twice, so I would like one of the charges corrected and the amount that was withdrawn to be reinstated please.' 'I got double charged!' 'I have a duplicate charge'
67	'what is the time frame for transfers' 'How long does it take transfers from Europe to complete?' 'If I need to transfer something quick from China, how long does that take?'
39	'Can I get a few more physical cards?' 'How do I obtain a second card?' 'Can my daughter also have a card issued in her name, but linked to my account?'
13	'Show me how to link the new card,' 'Where can I link a new card?' 'Can I reactivate my card? I reported it lost and found it this morning in my jacket.'
55	'cancel my account' 'I want to inactivate my account.' 'The service here is horrible, I want to delete my account.'
47	"My top-up hasn't gone through" 'Why is the top-up I made still pending.' "Hi, I tried topping up for the first time today (I'm a new customer), and the transaction has shown as pending for the last half an hour. Can you fix it?"
7	'My attempted transfer keeps kicking back with an error message. Could you please help me to resolve this?' 'I cannot make a transfer to another account.' "Why can't you allow my beneficiary?"
25	"Why won't you accept my payment? I keep trying with this card and it's not working." 'My payment for my card is not working' 'Please explain why my payment was declined? I thought everything was good.'
10	'Where do I pay with my card?' 'Can I use my card no matter where I go?' 'What businesses accept this card?'
64	'Why was money taken for a transfer?' 'Why do I see a fee when I do a transfer?' "I just recently got a new beach condo in Bora Bora, and to furnish it I used this account a lot from overseas. It's upsetting, being a long-time customer of yours, that I saw extra fees on my account from my currency transfer. Is this what I should expect from you in the future?"
35	'My transfer failed, why?' 'I needed to send my friends some money urgently. I tried multiple times to transfer the money this morning, but the transfer keeps failing. Can you tell me why the transfer is getting rejected?' "I have tried sending a standard transfer five times and it hasn't gone through. What is the problem?"
65	"I'd like to know how transferring my money into my account works." 'I would like to top up my account with a bank transfer, is that allowed?' 'How does your app transfer the money'
75	'Why did I get less cash than what I asked in the ATM?' "I'm at the ATM right now and it only allowed me to withdraw 10 pounds instead of my requested 30. What's going on? How can I fix this?" 'Hi, Please help me to request cash back. As i have been withdrawn money from ATM, but i got wrong amount. in Application its showing the amount i have been charged through.'
46	'There is a pending transaction on my account for when I tried to get money from the ATM earlier and it did not work. Can you fix this?' 'What does it mean if my cash withdrawal does not show up properly?' 'Why is my withdrawl to cash still pending?'
37	'Is there a way to use this app with a disposable virtual card?' 'Can you provide me a disposable virtual card?' 'How about disposable virtual cards?'
34	'There is a charge for 1£ on my statement. I do not think that I bought anything that costed 1£. What is going on?' 'There is an extra small charge on my card! What do I do?' "What am I being charged an extra pound when I haven't bought something worth that much?"
12	"I placed my order a week ago. Is it ok that my card isn't here yet?" 'I need my card delivered as soon as possible.' "Do I have a say when it's delivered?"
57	'Are there fees for using an international card to top up?' 'Do you have any fees if I want to add money using an international card.' 'What fees to you have for top ups?'
51	'There is not a refund in my statement.' "I'm expecting a refund but it's no where to be seen" "Hi! A seller that I requested a refund from a long time ago has yet to send the money into my account, even though I keep checking it. Can the seller be contacted by you guys to see what's going on?"
5	'How long do I have to wait for my bank transfer to appear in my account?' "I transferred some money a couple days ago but haven't received it yet. How long should I expect to wait?" 'I did a transfer but the balance is the same.'
9	'My card is expiring soon.' 'want a new card what are the fees' 'I think my card is expiring, what happens then?'
21	'How can I change my PIN? Help.' 'Are there cash machines where I can change my PIN?' 'Is there a location where I can change my PIN?'
76	'The wrong exchange rate was used for my cash withdrawal' 'Why did you give me so little currency?' 'I believe I got a wrong exchange rate when I got cash.'
36	'Are all European currencies available for holding and exchange?' 'What fiat currencies are used for holdings?' 'How can I solve my issue with holding multiple currencies?'
43	'Where do I order a card?' 'Do I have to pay for a physical card?' 'Where do I go to order my card?'
32	'What is the current exchange rate?' 'I need to know what your exchange rates are.' 'What are the exchange rates?'
20	'Somebody is taking money out of my account without my approval in another town separate from me. Please place a freeze on my account until I can make it to the bank.' "Someone stole my wallet a couple of hours ago now and they have taken money from my account. Is there any way that you can help so I won't lose any more of my money." 'There is random cash in my account.'
42	'Can my account me accessed by a thief? I was mugged.' 'my phone has been stolen by someone' 'My phone was left behind. Can I still use my card?'
68	'My identity verification is not working.' "It won't let me verify my identity." 'I need help proving that this is really me and to verify my identity.'
16	'Why is there an unknown card payment?' "I don't understand where this charge came from." "I didn't make a payment that shows in my app."
49	'I must have entered my PIN wrong.' 'My PIN was entered wrong and now I am blocked. Please unblock.' "My PIN can't access my card, can you help?."
1	'What is the minimum age required to open an account with your service?' 'Can I get an account for my daughter?' 'At what age can my children use your service?'
45	'How long will my payment be pending?' "Is there something wrong with my card payment? It's been pending a really, really long time." 'I used my card to make a payment a while ago and its still pending, Why is it taking so long?'
74	'What is identity check?' 'Can you explain more about why should I verify my identity?' 'Is identity verification required for me to use my account?'
56	'are SWIFT transfers an option?' 'Will there be a fee if I receive a SEPA transfer' 'If I top up by transfer, am I going to be charged?'
30	'How do I update my address?' "Why can't I edit my personal details?" "my identity hasn't been verified and i can't edit my details, what's next"
58	'Can I use cash to top up' "I'll send a check to top up my account" 'How do I use a cheque to top up?'
62	'I attempted to top up using my card, but now the money just disappeared?' 'Can you tell me how to top up on my card?' 'show me how to top up'
18	'My card is stuck inside the ATM, what am I supposed to do?' 'The ATM didnt return my card!' "The bank machine didn't give my card back how do I get a new one fast?"
54	'What currencies are approved to add money?' 'With what currencies can I add money?' 'I want to add money to my card, what payments do you accept?'
66	'How long will it take for my transfer to arrive?' 'I am having an issue with an in country transfer I did a few days ago. It has yet to appear in my account. I have checked to be sure that all account information is correct multiple times. What is taking the transfer so long?' "I'm still expecting the transaction to be finished"
19	'What is the extra fee I was charged when using an ATM?' 'Please help! There was an extra fee when I got cash.' 'I have a strange transaction on my account which appears to pertain to a cash withdrawal I made outside Nandos in Leeds. What is that for? Is it fraud?'
61	"Why isn't my top up showing? Was the money reversed?" 'I topped up but my money is not showing. Why is that?' 'It looks like my top-up was cancelled'
24	'I just moved to the US how do I get a card?' "I'm in the EU. Can I get one of your cards?" 'Are your cards available in the EU?'
69	'What do you need for the identity verification' 'What do I need to do to perform identity checks?' 'im worried about fraud how do you protect my account'
48	'When will my transfer move out of pending stage?' 'How long do I have to wait for the transfer to be finalized?' 'I need to know why a money transfer is available.'
6	'Why is it taking so long for my money to appear in my account? I have been waiting a week!' "The balance hasn't changed yet after depositing a cheque with some money." "I made a deposit a couple of days back it's, as of yet, not gone in! It would be ideal if you assist me with this"
8	'Hi, I made a transfer yesterday that I need to reverse. I need to put the money in a different account.' 'My transaction from earlier needs to be reversed.' 'I wish to cancel a transfer'
17	'my exchange rate looks wrong for my last payment' 'Please check my payment from last Saturday as I feel I have been overcharged on the exchange rate. Thank you.' 'I guess exchange rate for my payment was not correct, as i charged extra on the last saturday.'
53	"Hey a seller just contacted me that my money didn't come through. It was definitely deducted from my account, but now it appears it's back with me! Sort this out asap please." "I purchased something and already received it, why hasn't the money taken out?" 'I have cancelled payments to this card.'
2	'I got my American Express in Apple Pay, why is top up not working correctly?' 'Can I deposit money using Apple Pay?' 'Can I use my apple watch to top up?'
38	"I don't know my PIN, how can I find that out?" 'I dont have a pin' 'How do I set my physical card PIN?'
29	'Is there a transaction limit on a disposable card?' 'I have to make more then a few disposable cards per day.' 'What number of transactions can i do with one disposable card?'
71	'How do I verify the top-up card?' 'Why should I verify the top-up?' 'What is the significance of verifying the top-up?'
31	'Any extra charges for exchanging currencies?' 'Are there any fees when exchanging to foreign currencies?' 'Are there any discounts for frequent currency exchanges?'

Evaluation

Metrics

Label	Accuracy	F1_Score	Recall	Precision
all	0.8964	0.8962	0.8964	0.9020

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("chrischak66/gte-multilingual-base-banking77-setfit-5-v3")
# Run inference
preds = model("Where can I use my card?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	11.5678	78

Label	Training Sample Count
0	25
1	25
2	25
3	25
4	25
5	25
6	25
7	25
8	25
9	25
10	25
11	25
12	25
13	25
14	25
15	25
16	25
17	25
18	25
19	25
20	25
21	25
22	25
23	25
24	25
25	25
26	25
27	25
28	25
29	25
30	25
31	25
32	25
33	25
34	25
35	25
36	25
37	25
38	25
39	25
40	25
41	25
42	25
43	25
44	25
45	25
46	25
47	25
48	25
49	25
50	25
51	25
52	25
53	25
54	25
55	25
56	25
57	25
58	25
59	25
60	25
61	25
62	25
63	25
64	25
65	25
66	25
67	25
68	25
69	25
70	25
71	25
72	25
73	25
74	25
75	25
76	25

Training Hyperparameters

• batch_size: (64, 64)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• body_learning_rate: (5e-05, 5e-05)
• head_learning_rate: 0.01
• loss: BatchHardTripletLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0323	1	0.1275	-
0.3226	10	0.1297	-
0.6452	20	0.1263	-
0.9677	30	0.1264	-
1.0	31	-	0.1107

Framework Versions

• Python: 3.10.12
• SetFit: 1.1.0
• Sentence Transformers: 3.2.1
• Transformers: 4.42.2
• PyTorch: 2.5.1+cu121
• Datasets: 3.1.0
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}