google/seahorse-large-q1

This is model based on mT5-L that predicts a binary label for a given article and summary for Q1 (comprehensibility), as defined in the SEAHORSE paper (Clark et al., 2023).

It is trained similarly to the TRUE paper (Honovich et al, 2022) on human ratings from the SEAHORSE dataset in 6 languages:

• German
• English
• Spanish
• Russian
• Turkish
• Vietnamese

The input format for the model is: "premise: ARTICLE hypothesis: SUMMARY", where ARTICLE is the document being summarized and SUMMARY is the candidate summary.

There is also an XXL version of this model, as well as metrics trained for each of the other 5 dimensions described in the original paper.

The full citation for the SEAHORSE paper is:

@misc{clark2023seahorse,
      title={SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation}, 
      author={Elizabeth Clark and Shruti Rijhwani and Sebastian Gehrmann and Joshua Maynez and Roee Aharoni and Vitaly Nikolaev and Thibault Sellam and Aditya Siddhant and Dipanjan Das and Ankur P. Parikh},
      year={2023},
      eprint={2305.13194},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contact: seahorse-authors@google.com