|
--- |
|
metrics: |
|
- Recall @10 0.438 |
|
- MRR @10 0.247 |
|
base_model: |
|
- unicamp-dl/mt5-base-mmarco-v2 |
|
tags: |
|
- Information Retrieval |
|
- Natural Language Processing |
|
- Question Answering |
|
license: apache-2.0 |
|
--- |
|
|
|
# Urdu-mT5-mmarco: Fine-Tuned mT5 Model for Urdu Information Retrieval |
|
|
|
As part of ongoing efforts to make Information Retrieval (IR) more inclusive, this model addresses the needs of low-resource languages, focusing specifically on Urdu. |
|
We created this model by translating the MS-Marco dataset into Urdu using the IndicTrans2 model. |
|
To establish baseline performance, we initially tested for zero-shot learning for IR in Urdu using the unicamp-dl/mt5-base-mmarco-v2 model |
|
and then applied fine-tuning with the mMARCO multilingual IR methodology on the translated dataset. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
|
|
|
|
- **Developed by:** Umer Butt |
|
- **Model type:** MT5ForConditionalGeneration |
|
- **Language(s) (NLP):** Python/pytorch |
|
|
|
|
|
|
|
## Uses |
|
|
|
|
|
|
|
### Direct Use |
|
|
|
|
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
Although this model performs well and is state-of-the-art for now. But still this model is finetuned on mmarco model and a translated dataset(which was created using indicTrans2 model). Hence the limitations of those apply here too. |
|
|
|
### Recommendations |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
Example Code for Scoring Query-Document Pairs: |
|
In an IR setting, you provide a query and one or more candidate documents. The model scores each document for relevance to the query, which can be used for ranking. |
|
``` |
|
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
|
import torch |
|
|
|
# Load the tokenizer and model |
|
tokenizer = AutoTokenizer.from_pretrained("Mavkif/urdu-mt5-mmarco") |
|
model = AutoModelForSeq2SeqLM.from_pretrained("Mavkif/urdu-mt5-mmarco") |
|
|
|
# Define the query and candidate documents |
|
query = "پاکستان کی معیشت کی موجودہ صورتحال کیا ہے؟" |
|
document_1 = "پاکستان کی معیشت میں حالیہ ترقی کے بارے میں معلومات۔" |
|
document_2 = "فٹبال پاکستان میں تیزی سے مقبول ہو رہا ہے۔" |
|
|
|
# Tokenize query-document pairs and calculate relevance scores |
|
def get_score(query, document): |
|
input_text = f"Query: {query} Document: {document}" |
|
inputs = tokenizer(input_text, return_tensors="pt", truncation=True) |
|
|
|
# Pass through the model and get the relevance score (logits) |
|
outputs = model(**inputs) |
|
score = outputs.logits[0, -1, :] # last token logits |
|
return torch.softmax(score, dim=0)[tokenizer.eos_token_id].item() |
|
|
|
# Get scores for each document |
|
score_1 = get_score(query, document_1) |
|
score_2 = get_score(query, document_2) |
|
|
|
print(f"Relevance Score for Document 1: {score_1}") |
|
print(f"Relevance Score for Document 2: {score_2}") |
|
|
|
# Higher score indicates higher relevance |
|
|
|
``` |
|
|
|
|
|
|
|
## Evaluation |
|
|
|
The evaluation was done using the scripts in the pygaggle library. Specifically these files: |
|
evaluate_monot5_reranker.py |
|
ms_marco_eval.py |
|
|
|
#### Metrics |
|
Following the approach in the mmarco work. The same two metrics were used. |
|
|
|
Recal @10 : 0.438 |
|
MRR @10 : 0.247 |
|
|
|
|
|
### Results |
|
|
|
| Model | Name | Data | Recall@10 | MRR@10 | Queries Ranked | |
|
|---------------------------------------|---------------------------------------|--------------|-----------|--------|----------------| |
|
| bm25 (k = 1000) | BM25 - Baseline from mmarco paper | English data | 0.391 | 0.187 | 6980 | |
|
| unicamp-dl/mt5-base-mmarco-v2 | mmarco reranker - Baseline from paper | English data | | 0.370 | 6980 | |
|
| bm25 (k = 1000) | BM25 | Urdu data | 0.2675 | 0.129 | 6980 | |
|
| unicamp-dl/mt5-base-mmarco-v2 | Zero-shot mmarco | Urdu data | 0.408 | 0.204 | 6980 | |
|
| This work | Mavkif/urdu-mt5-mmarco | Urdu data | 0.438 | 0.247 | 6980 | |
|
|
|
|
|
|
|
|
|
|
|
### Model Architecture and Objective |
|
{ |
|
"_name_or_path": "unicamp-dl/mt5-base-mmarco-v2", |
|
"architectures": ["MT5ForConditionalGeneration"], |
|
"d_model": 768, |
|
"num_heads": 12, |
|
"num_layers": 12, |
|
"dropout_rate": 0.1, |
|
"vocab_size": 250112, |
|
"model_type": "mt5", |
|
"transformers_version": "4.38.2" |
|
} |
|
For more details on how to customize the decoding parameters (such as max_length, num_beams, and early_stopping), refer to the Hugging Face documentation. |
|
|
|
|
|
## Model Card Authors [optional] |
|
|
|
Umer Butt |
|
|
|
|
|
## Model Card Contact |
|
|
|
mumertbutt@gmail.com |
