Tos-Roberta: Terms of Service Fairness Classifier

Model Description

Tos-Roberta is a fine-tuned RoBERTa-large model designed to classify clauses in Terms of Service (ToS) documents based on their fairness level. The model categorizes clauses into three classes: clearly fair, potentially unfair, and clearly unfair.

Task

The model performs multi-class classification on individual sentences or clauses, categorizing them into three levels of unfairness:

0. Clearly Fair
1. Potentially Unfair
2. Clearly Unfair

Key Features

• Based on the RoBERTa-large architecture
• Fine-tuned on a specialized dataset of ToS clauses
• Achieves high accuracy in distinguishing between fair and unfair clauses
• Suitable for legal text analysis and consumer rights applications

Performance

The model demonstrates strong performance on the task of ToS clause classification:

• Validation Accuracy: 89.64%
• Test Accuracy: 85.84%

Detailed performance metrics per epoch:

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score	Precision	Recall
1	0.443500	0.398950	0.874699	0.858838	0.862516	0.874699
2	0.416400	0.438409	0.853012	0.847317	0.849916	0.853012
3	0.227700	0.505879	0.896386	0.893325	0.891521	0.896386
4	0.052600	0.667532	0.891566	0.893167	0.895115	0.891566
5	0.124200	0.747090	0.884337	0.887412	0.891807	0.884337

Training Details

• Base Model: RoBERTa-large
• Dataset: CodeHima/TOS_DatasetV2
• Training Time: 3310.09 seconds
• Epochs: 5
• Batch Size: 8
• Learning Rate: Started at 2e-5 with a warmup period and decay
• Optimizer: AdamW
• Loss Function: Cross-Entropy
• Training Strategy: Mixed precision training (FP16)

Usage

To use this model for inference:

from transformers import RobertaTokenizer, RobertaForSequenceClassification
import torch

# Load model and tokenizer
model = RobertaForSequenceClassification.from_pretrained("YourHuggingFaceUsername/Tos-Roberta")
tokenizer = RobertaTokenizer.from_pretrained("YourHuggingFaceUsername/Tos-Roberta")

# Prepare input text
text = "Your Terms of Service clause here"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=512)

# Make prediction
with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(outputs.logits, dim=-1)
predicted_class = torch.argmax(probabilities, dim=-1).item()

# Map prediction to label
label_map = {0: "clearly_fair", 1: "potentially_unfair", 2: "clearly_unfair"}
predicted_label = label_map[predicted_class]

print(f"Predicted class: {predicted_label}")
print(f"Probabilities: {probabilities[0].tolist()}")

Limitations and Bias

• The model's performance may vary depending on the legal jurisdiction and specific domain of the ToS.
• It may not capture nuanced legal interpretations that require human expertise.
• The training data may contain biases present in existing ToS documents.

Ethical Considerations

While this model can assist in identifying potentially unfair clauses in ToS documents, it should not be used as a substitute for professional legal advice. The model's predictions should be reviewed by qualified legal professionals before making any decisions based on its output.

Citation

If you use this model in your research or application, please cite it as:

@misc{Tos-Roberta,
  author = {Himanshu Mohanty},
  title = {Tos-Roberta: RoBERTa-large model for Terms of Service Fairness Classification},
  year = {2024},
  publisher = {HuggingFace},
  journal = {CodeHima/Tos-Roberta},
  howpublished = {\url{https://huggingface.co/CodeHima/Tos-Roberta}}
}

Contact

For questions, feedback, or collaborations, please open an issue on the model's Hugging Face repository or contact [Your Contact Information].