README.md

metadata

license: cc-by-nc-4.0
language:
  - ar
  - az
  - bg
  - de
  - el
  - en
  - es
  - fr
  - hi
  - it
  - ja
  - nl
  - pl
  - pt
  - ru
  - sw
  - th
  - tr
  - ur
  - vi
  - zh
pipeline_tag: text-classification
tags:
  - language detect

Multilingual Language Detection Model

Model Description

This repository contains a multilingual language detection model based on the XLM-RoBERTa base architecture. The model is capable of distinguishing between 21 different languages including Arabic, Azerbaijani, Bulgarian, German, Greek, English, Spanish, French, Hindi, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swahili, Thai, Turkish, Urdu, Vietnamese, and Chinese.

How to Use

You can use this model directly with a pipeline for text classification, or you can use it with the transformers library for more custom usage, as shown in the example below.

Quick Start

First, install the transformers library if you haven't already:

pip install transformers


from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("LocalDoc/language_detection")
model = AutoModelForSequenceClassification.from_pretrained("LocalDoc/language_detection")

# Prepare text
text = "Əlqasım oğulları vorzakondu"
encoded_input = tokenizer(text, return_tensors='pt', truncation=True, max_length=512)

# Prediction
model.eval()
with torch.no_grad():
    outputs = model(**encoded_input)

# Process the outputs
logits = outputs.logits
probabilities = torch.nn.functional.softmax(logits, dim=-1)
predicted_class_index = probabilities.argmax().item()
labels = ["az", "ar", "bg", "de", "el", "en", "es", "fr", "hi", "it", "ja", "nl", "pl", "pt", "ru", "sw", "th", "tr", "ur", "vi", "zh"]
predicted_label = labels[predicted_class_index]
print(f"Predicted Language: {predicted_label}")


Training Performance

The model was trained over three epochs, showing consistent improvement in accuracy and loss:

    <b>Epoch 1:</b> Training Loss: 0.0127, Validation Loss: 0.0174, Accuracy: 0.9966, F1 Score: 0.9966
    <b>Epoch 2:</b> Training Loss: 0.0149, Validation Loss: 0.0141, Accuracy: 0.9973, F1 Score: 0.9973
    <b>Epoch 3:</b> Training Loss: 0.0001, Validation Loss: 0.0109, Accuracy: 0.9984, F1 Score: 0.9984

Test Results

The model achieved the following results on the test set:

    Loss: 0.0133
    Accuracy: 0.9975
    F1 Score: 0.9975
    Precision: 0.9975
    Recall: 0.9975
    Evaluation Time: 17.5 seconds
    Samples per Second: 599.685
    Steps per Second: 9.424

Licensing

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. You are free to use, modify, and distribute this model non-commercially, provided you attribute the original creation.