IsmatS/xlm_roberta_large_az_ner

Azerbaijani Named Entity Recognition (NER) with XLM-RoBERTa Large

Repository on Hugging Face: IsmatS/xlm_roberta_large_az_ner
Repository on GitHub: Named Entity Recognition

File Structure

.
├── README.md                   # Documentation for the project
├── config.json                 # Configuration file for model deployment
├── model-001.safetensors       # Model weights in Safetensors format for safe deployment
├── sentencepiece.bpe.model     # SentencePiece model for tokenization
├── special_tokens_map.json     # Map for special tokens (e.g., <PAD>, <CLS>)
├── tokenizer.json              # JSON configuration for tokenizer
├── tokenizer_config.json       # Additional tokenizer configurations
├── xlm_roberta_large.ipynb     # Jupyter Notebook for training and experimentation
└── xlm_roberta_large.py        # Python script for training and experimentation

Explanation:

• README.md: Provides detailed information on the project, including setup, usage, and evaluation.
• config.json: Stores configuration details for model deployment, such as model parameters.
• model-001.safetensors: Contains model weights in a secure, efficient format.
• sentencepiece.bpe.model: Tokenization model used to segment sentences into subwords for xlm-roberta-large.
• special_tokens_map.json: Maps special tokens required by the tokenizer (e.g., <PAD> for padding).
• tokenizer.json: Contains the main tokenizer configuration.
• tokenizer_config.json: Additional configuration settings for the tokenizer.
• xlm_roberta_large.ipynb: A Jupyter notebook for experimenting with and training the model.
• xlm_roberta_large.py: Python script for training and running evaluations outside of Jupyter.

Project Overview

This project leverages xlm-roberta-large, a multilingual transformer model, fine-tuned for Azerbaijani Named Entity Recognition (NER). The model identifies various named entities, including persons, organizations, dates, etc., using a dataset specially designed for the Azerbaijani language.

Table of Contents

1. Setup and Dependencies
2. Dataset
3. Model Architecture
4. Training Process
5. Training Metrics and Results
6. Evaluation and Detailed Metrics Explanation
7. Saving and Loading the Model
8. Example Inference
9. Troubleshooting and Notes

---

Setup and Dependencies

Install the required libraries:

!pip install transformers datasets seqeval huggingface_hub

Imports

The project requires transformers, datasets, torch, and seqeval libraries.

Dataset

The Azerbaijani NER dataset includes entities such as PERSON, LOCATION, ORGANISATION, etc., and is hosted on Hugging Face.

from datasets import load_dataset
dataset = load_dataset("LocalDoc/azerbaijani-ner-dataset")

A preprocessing function ensures token and NER tag formatting:

def preprocess_example(example):
    example["tokens"] = ast.literal_eval(example["tokens"])
    example["ner_tags"] = list(map(int, ast.literal_eval(example["ner_tags"])))
    return example

dataset = dataset.map(preprocess_example)

Model Architecture

The model is based on xlm-roberta-large, designed to handle multilingual text processing.

Tokenization and Label Alignment

A custom function tokenize_and_align_labels tokenizes input while aligning entity labels with tokens.

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")

def tokenize_and_align_labels(example):
    tokenized_inputs = tokenizer(
        example["tokens"], truncation=True, is_split_into_words=True,
        padding="max_length", max_length=128
    )
    # Alignment code follows here...
    return tokenized_inputs

---

Training Process

Training uses the Hugging Face Trainer, which handles the training loop, metrics computation, and model checkpointing.

Model Initialization

from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained(
    "xlm-roberta-large", num_labels=len(label_list)
)

Define Evaluation Metrics

The following metrics help evaluate the model’s accuracy and performance in recognizing and classifying entities.

from seqeval.metrics import precision_score, recall_score, f1_score, classification_report

def compute_metrics(p):
    # Metric computation code
    return {
        "precision": precision_score(true_labels, true_predictions),
        "recall": recall_score(true_labels, true_predictions),
        "f1": f1_score(true_labels, true_predictions),
    }

Training Configuration

training_args = TrainingArguments(
    output_dir="./results", evaluation_strategy="epoch", save_strategy="epoch",
    learning_rate=2e-5, per_device_train_batch_size=128, per_device_eval_batch_size=128,
    num_train_epochs=12, weight_decay=0.005, fp16=True, logging_dir='./logs',
    save_total_limit=2, load_best_model_at_end=True, metric_for_best_model="f1", report_to="none"
)

---

Training Metrics and Results

During training, metrics for each epoch were recorded. These metrics include Training Loss, Validation Loss, Precision, Recall, and F1-Score for both training and validation sets.

Epoch	Training Loss	Validation Loss	Precision	Recall	F1-Score
1	0.4075	0.2538	0.7689	0.7214	0.7444
2	0.2556	0.2497	0.7835	0.7245	0.7528
3	0.2144	0.2488	0.7509	0.7489	0.7499
4	0.1934	0.2571	0.7686	0.7404	0.7542
5	0.1698	0.2757	0.7458	0.7537	0.7497
6	0.1526	0.2881	0.7831	0.7284	0.7548
7	0.1443	0.3034	0.7585	0.7381	0.7481
8	0.1268	0.3113	0.7456	0.7509	0.7482
9	0.1194	0.3316	0.7393	0.7495	0.7444
10	0.1094	0.3448	0.7543	0.7372	0.7456
11	0.1029	0.3549	0.7519	0.7413	0.7466

These metrics demonstrate the model’s performance progression through each epoch, highlighting how it optimizes towards lower losses and higher precision, recall, and F1-scores.

---

Evaluation and Detailed Metrics Explanation

After training, the model was evaluated across various entity types with the following results:

Entity	Precision	Recall	F1-score	Support
ART	0.41	0.19	0.26	1828
DATE	0.53	0.49	0.51	834
EVENT	0.67	0.51	0.58	63
FACILITY	0.74	0.68	0.71	1134
LAW	0.62	0.58	0.60	1066
LOCATION	0.81	0.79	0.80	8795
MONEY	0.59	0.56	0.58	555
ORGANISATION	0.70	0.69	0.70	554
PERCENTAGE	0.80	0.82	0.81	3502
PERSON	0.90	0.82	0.86	7007
PRODUCT	0.83	0.84	0.84	2624
TIME	0.60	0.53	0.57	1584

Explanation of Metrics

• Precision: Represents the proportion of correctly identified entities out of all entities predicted by the model. High precision is vital in NER tasks to reduce false positives, ensuring only actual entities are classified.
• Recall: Indicates the proportion of correctly identified entities out of all actual entities present in the dataset. Higher recall captures all relevant entities, which is essential to avoid missing important information.
• F1-Score: The harmonic mean of precision and recall, balancing both metrics. A high F1-score suggests an effective trade-off between precision and recall, ideal for NER where both metrics are crucial for accurate entity recognition.

Saving and Loading the Model

Save the model and tokenizer for future use or further fine-tuning:

save_directory = "./xlm-roberta-large"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

---

Example Inference

Load the saved model for inference, utilizing the Hugging Face pipeline for NER.

from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained(save_directory)
model = AutoModelForTokenClassification.from_pretrained(save_directory)
device = 0 if torch.cuda.is_available() else -1
nlp_ner = pipeline("ner", model=model, tokenizer=tokenizer

, aggregation_strategy="simple", device=device)

# Example sentence
test_texts = ["Shahla Khuduyeva və Pasha Sığorta şirkəti haqqında məlumat."]
evaluate_model(test_texts, [["B-PERSON", "B-ORGANISATION"]])

---

Troubleshooting and Notes

• Token Alignment: Tokenization and label alignment must be carefully handled, especially when dealing with subwords.
• Memory Management: Adjust batch size if encountering GPU memory limitations.
• Early Stopping: Configured with a patience of 5 epochs to avoid overfitting, automatically halting training if validation performance plateaus.

This README provides detailed information on the project setup, training, evaluation, and inference for the xlm-roberta-large NER model fine-tuned for Azerbaijani text. The model's performance on various entity types and the clear explanation of metrics make it a comprehensive resource for Azerbaijani NER tasks.