metadata

language: multilingual
thumbnail: null

A fine-tuned model on GoldP task from Tydi QA dataset

This model uses bert-multi-cased-finetuned-xquadv1 and fine-tuned on Tydi QA dataset for Gold Passage task (GoldP)

Details of the language model

The base language model (bert-multi-cased-finetuned-xquadv1) is a fine-tuned version of bert-base-multilingual-cased for the Q&A downstream task

Details of the Tydi QA dataset

TyDi QA contains 200k human-annotated question-answer pairs in 11 Typologically Diverse languages, written without seeing the answer and without the use of translation, and is designed for the training and evaluation of automatic question answering systems. This repository provides evaluation code and a baseline system for the dataset. https://ai.google.com/research/tydiqa

Details of the downstream task (Gold Passage or GoldP aka the secondary task)

Given a passage that is guaranteed to contain the answer, predict the single contiguous span of characters that answers the question. the gold passage task differs from the primary task in several ways:

only the gold answer passage is provided rather than the entire Wikipedia article;
unanswerable questions have been discarded, similar to MLQA and XQuAD;
we evaluate with the SQuAD 1.1 metrics like XQuAD; and
Thai and Japanese are removed since the lack of whitespace breaks some tools.

Model training

The model was fine-tuned on a Tesla P100 GPU and 25GB of RAM. The script is the following:

python run_squad.py \
  --model_type bert \
  --model_name_or_path mrm8488/bert-multi-cased-finetuned-xquadv1 \
  --do_train \
  --do_eval \
  --train_file /content/dataset/train.json \
  --predict_file /content/dataset/dev.json \
  --per_gpu_train_batch_size 24 \
  --per_gpu_eval_batch_size 24 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.5 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --overwrite_output_dir \
  --save_steps 5000 \
  --threads 40

Global Results (dev set):

Metric	# Value
Exact	71.06
F1	82.16

Specific Results (per language):

Language	# Samples	# Exact	# F1
Arabic	1314	73.29	84.72
Bengali	180	64.60	77.84
English	654	72.12	82.24
Finnish	1031	70.14	80.36
Indonesian	773	77.25	86.36
Korean	414	68.92	70.95
Russian	1079	62.65	78.55
Swahili	596	80.11	86.18
Telegu	874	71.00	84.24

Created by Manuel Romero/@mrm8488

Made with ♥ in Spain