Model card for PM-AI/paraphrase-distilroberta-base-v2_de-en

For internal purposes and for testing, we have made a monolingual paraphrasing model from Sentence Transformers usable for German + English via Knowledge Distillation. The decision was made in favor of sentence-transformers/paraphrase-distilroberta-base-v2 because this model has no public available multilingual version (to our knowledge). In addition, it has significantly more training samples compared to its predecessor: 83.3 million samples were used instead of 24.6 million samples.

Training

Download of datasets
Execution of knowledge distillation

Training Data

Datasets used based on offical source:

AllNLI
sentence-compression
SimpleWiki
altlex
msmarco-triplets
quora_duplicates
coco_captions
flickr30k_captions
yahoo_answers_title_question
S2ORC_citation_pairs
stackexchange_duplicate_questions
wiki-atomic-edits

Training Execution

First we downloaded some german-english parallel datasets via get_parallel_data_*.py.

These datasets are: Tatoeba, WikiMatrix, TED2020, OpenSubtitles, Europarl, News-Commentary

Then we started knowledge distillation with make_multilingual_sys.py

Parameterization of training

Script: make_multilingual_sys.py
Datasets: Tatoeba, WikiMatrix, TED2020, OpenSubtitles, Europarl, News-Commentary
GPU: NVIDIA A40 (Driver Version: 515.48.07; CUDA Version: 11.7)
Batch Size: 64
Max Sequence Length: 256
Train Max Sentence Length: 600
Max Sentences Per Train File: 1000000
Teacher Model: sentence-transformers/paraphrase-distilroberta-base-v2
Student Model: xlm-roberta-base
Loss Function: MSE Loss
Learning Rate: 2e-5
Epochs: 20
Evaluation Steps: 10000
Warmup Steps: 10000

Acknowledgment

This work is a collaboration between Technical University of Applied Sciences Wildau (TH Wildau) and sense.ai.tion GmbH. You can contact us via:

Philipp Müller (M.Eng.); Author
Prof. Dr. Janett Mohnke; TH Wildau
Dr. Matthias Boldt, Jörg Oehmichen; sense.AI.tion GmbH

This work was funded by the European Regional Development Fund (EFRE) and the State of Brandenburg. Project/Vorhaben: "ProFIT: Natürlichsprachliche Dialogassistenten in der Pflege".