metadata
'[object Object]': null
license: cc
language:
- en
library_name: adapter-transformers
pipeline_tag: text-classification
Model Card for orYx-models/finetuned-roberta-leadership-sentiment-analysis
- Model Description: This model is a finetuned version of the RoBERTa text classifier(cardiffnlp/twitter-roberta-base-sentiment-latest). It has been trained on a dataset comprising communications from corporate executives to their therapists. Its primary function is to determine whether statements from corporate executives convey a "Positive," "Negative," or "Neutral" sentiment, accompanied by a confidence level indicating the percentage of sentiment expressed in a statement. Being a prototype tool by orYx Models, all feedbacks and insights will be used to further refine the model.
Model Details
Model Information
- Model Type: Text Classifier
- Language(s): English
- License: Creative Commons license family
- Finetuned from Model: cardiffnlp/twitter-roberta-base-sentiment-latest
Model Sources
- HuggingFace Model ID: cardiffnlp/twitter-roberta-base-2021-124m
- Paper: TimeLMs - Link
Uses
- Use case: This sentiment analysis tool can analyze text from any user within an organization, such as executives, employees, or clients, and assign a sentiment to it.
- Outcomes: The tool generates a "Scored sentiment" which can be used to assess the likelihood of events occurring or vice versa. It can also facilitate the creation of a rating system based on the sentiments expressed in texts.
Direct Use
nlp = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)
nlp("The results don't match, but the effort seems to be always high")
Out[7]: [{'label': 'Positive', 'score': 0.9996090531349182}]
- Based on the text the outcomes can be "Positive, Negative, Neutral" along with their confidence score.
Recommendations
- Continuous Monitoring: Regularly monitor the model's performance on new data to ensure its effectiveness and reliability over time.
- Error Analysis: Conduct thorough error analysis to identify common patterns of misclassifications and areas for improvement.
- Fine-Tuning: Consider fine-tuning the model further based on feedback and insights from users, to enhance its domain-specific performance.
- Model Interpretability: Explore techniques for explaining the model's predictions, such as attention mechanisms or feature importance analysis, to increase trust and understanding of its decisions.
Training Details
X_train, X_val, y_train, y_val = train_test_split(X,y, test_size = 0.2, stratify = y)
- Train data: 80% of 4396 records = 3516
- Test data: 20% of 4396 records = 879
Training Procedure
- Dataset Split: Data divided into 80% training and 20% validation sets.
- Preprocessing: Input data tokenized into 'input_ids' and 'attention_mask' tensors.
- Training Hyperparameters: Set for training, evaluation, and optimization, including batch size, epochs, and logging strategies.
- Training Execution: Model trained with specified hyperparameters, monitored with metrics, and logged for evaluation.
- Evaluation Metrics: Model evaluated on loss, accuracy, F1 score, precision, and recall for both training and validation sets.
Preprocessing [optional]
'input_ids': tensor
'attention_mask': tensor
'label': tensor(2)
Training Hyperparameters
args = TrainingArguments(
output_dir="output",
do_train = True,
do_eval = True,
num_train_epochs = 1,
per_device_train_batch_size = 4,
per_device_eval_batch_size = 8,
warmup_steps = 50,
weight_decay = 0.01,
logging_strategy= "steps",
logging_dir= "logging",
logging_steps = 50,
eval_steps = 50,
save_strategy = "steps",
fp16 = True,
#load_best_model_at_end = True
)
Speeds, Sizes, Times [optional]
- TrainOutput
global_step=879,
training_loss=0.1825900522650848,
- Metrics
'train_runtime': 101.6309,
'train_samples_per_second': 34.596,
'train_steps_per_second': 8.649,
'total_flos': 346915041274368.0,
'train_loss': 0.1825900522650848,
'epoch': 1.0
Evaluation Metrics Results
# Assuming you have a list of evaluation results q and want to create a DataFrame with it
q = [Trainer.evaluate(eval_dataset=df) for df in [train_dataset, val_dataset]]
# Create DataFrame with index and select only the first 5 columns
result_df = pd.DataFrame(q, index=["train", "val"]).iloc[:,:5]
# Display the resulting DataFrame
print(result_df)
______________________________________________________________________
eval_loss eval_Accuracy eval_F1 eval_Precision eval_Recall
train 0.049349 0.988908 0.987063 0.982160 0.992357
val 0.108378 0.976136 0.972464 0.965982 0.979861
______________________________________________________________________
loss
- train 0.049349
- val 0.108378
Accuracy
- train 0.988908 - 98.8%
- val 0.976136 - 97.6%
F1
- train 0.987063 - 98.7%
- val 0.972464 - 97.2%
Precision
- train 0.982160 - 98.2%
- val 0.965982 - 96.5%
Recall
- train 0.992357 - 99.2%
- val 0.979861 - 97.9%
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: T4 GPU
- Hours used: 2
- Cloud Provider: Google
- Compute Region: India
- Carbon Emitted: No Information Available
Compute Infrastructure
Google Colab - T4 GPU
References
@inproceedings{camacho-collados-etal-2022-tweetnlp,
title = "{T}weet{NLP}: Cutting-Edge Natural Language Processing for Social Media",
author = "Camacho-collados, Jose and
Rezaee, Kiamehr and
Riahi, Talayeh and
Ushio, Asahi and
Loureiro, Daniel and
Antypas, Dimosthenis and
Boisson, Joanne and
Espinosa Anke, Luis and
Liu, Fangyu and
Mart{\'\i}nez C{\'a}mara, Eugenio" and others,
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = dec,
year = "2022",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-demos.5",
pages = "38--49"
}
Model Card Authors [optional]
Vineedhar, relkino