Math-aware RoBERTa

This repository contains our pre-trained RoBERTa-based model for ARQMath 3. It was initialised from RoBERTa-base and further pre-trained on Math StackExchange in only one stage. We also added more LaTeX tokens to the tokenizer to enable a better tokenization of mathematical formulas. This model is not yet fine-tuned on a specific task.

For further details, please read our paper: http://ceur-ws.org/Vol-3180/paper-07.pdf.

Usage

You can use this model to further fine-tune it on any math-aware task you have in mind, e.g., classification, question-answering, etc. . Please note, that the model in this repository is only pre-trained and not fine-tuned.

Citation

If you find this model useful, consider citing our paper:

@article{reusch2022transformer,
  title={Transformer-Encoder and Decoder Models for Questions on Math},
  author={Reusch, Anja and Thiele, Maik and Lehner, Wolfgang},
  year={2022},
  organization={CLEF}
}