BERT-based Organization Detection Model for Turkish Texts

Model Description

This model is fine-tuned on the dbmdz/bert-base-turkish-uncased architecture for detecting organization accounts within Turkish Twitter. This initiative is part of the Politus Project's efforts to analyze organizational presence in social media data.

Model Architecture

Base Model: BERT (dbmdz/bert-base-turkish-uncased)
Training Data: Twitter data from 4,000 random accounts and 12,000 accounts with high organization-related activity as determined by m3inference scores above 0.7, 8,000 accounts in total. The data was annotated based on user names, screen names, and descriptions using ChatGPT 4. Positives are checked by hand.

Training Setup

Tokenization: Used Hugging Face's AutoTokenizer, padding sequences to a maximum length of 128 tokens.
Dataset Split: 80% training, 20% validation.
Training Parameters:
- Epochs: 3
- Training batch size: 8
- Evaluation batch size: 16
- Warmup steps: 500
- Weight decay: 0.01

Hyperparameter Tuning

Performed using Optuna, achieving best settings:

Learning rate: 3.2167327348605205e-05
Batch size: 16
Epochs: 1

Evaluation Metrics

Precision on Validation Set: 0.76 (organization class)
Recall on Validation Set: 0.84 (organization class)
F1-Score (Macro Average): 0.79
Accuracy: 0.95
Confusion Matrix on Validation Set:

[[2715, 103],
[ 60, 322]]

Hand-coded Sample of 1000 Accounts:
Precision: 0.89
Recall: 0.89
F1-Score (Macro Average): 0.89
Confusion Matrix:
```
[[936, 3],
 [ 4, 31]]
```

How to Use

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("atsizelti/atsizelti/turkish_org_classifier_16k")
tokenizer = AutoTokenizer.from_pretrained("atsizelti/atsizelti/turkish_org_classifier_16k")

text = "Örnek metin buraya girilir."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
predictions = outputs.logits.argmax(-1)