metadata

title: Competition MATH
emoji: 🤗
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
tags:
  - evaluate
  - metric
description: >-
  This metric is used to assess performance on the Mathematics Aptitude Test of
  Heuristics (MATH) dataset. It first canonicalizes the inputs (e.g., converting
  "1/2" to "\frac{1}{2}") and then computes accuracy.

Metric Card for Competition MATH

Metric description

This metric is used to assess performance on the Mathematics Aptitude Test of Heuristics (MATH) dataset.

It first canonicalizes the inputs (e.g., converting 1/2 to \\frac{1}{2}) and then computes accuracy.

How to use

This metric takes two arguments:

predictions: a list of predictions to score. Each prediction is a string that contains natural language and LaTeX.

references: list of reference for each prediction. Each reference is a string that contains natural language and LaTeX.

>>> from evaluate import load
>>> math = load("competition_math")
>>> references = ["\\frac{1}{2}"]
>>> predictions = ["1/2"]
>>> results = math.compute(references=references, predictions=predictions)

N.B. To be able to use Competition MATH, you need to install the math_equivalence dependency using pip install git+https://github.com/hendrycks/math.git.

Output values

This metric returns a dictionary that contains the accuracy after canonicalizing inputs, on a scale between 0.0 and 1.0.

Values from popular papers

The original MATH dataset paper reported accuracies ranging from 3.0% to 6.9% by different large language models.

More recent progress on the dataset can be found on the dataset leaderboard.

Examples

Maximal values (full match):

>>> from evaluate import load
>>> math = load("competition_math")
>>> references = ["\\frac{1}{2}"]
>>> predictions = ["1/2"]
>>> results = math.compute(references=references, predictions=predictions)
>>> print(results)
{'accuracy': 1.0}

Minimal values (no match):

>>> from evaluate import load
>>> math = load("competition_math")
>>> references = ["\\frac{1}{2}"]
>>> predictions = ["3/4"]
>>> results = math.compute(references=references, predictions=predictions)
>>> print(results)
{'accuracy': 0.0}

Partial match:

>>> from evaluate import load
>>> math = load("competition_math")
>>> references = ["\\frac{1}{2}","\\frac{3}{4}"]
>>> predictions = ["1/5", "3/4"]
>>> results = math.compute(references=references, predictions=predictions)
>>> print(results)
{'accuracy': 0.5}

Limitations and bias

This metric is limited to datasets with the same format as the Mathematics Aptitude Test of Heuristics (MATH) dataset, and is meant to evaluate the performance of large language models at solving mathematical problems.

N.B. The MATH dataset also assigns levels of difficulty to different problems, so disagregating model performance by difficulty level (similarly to what was done in the original paper can give a better indication of how a given model does on a given difficulty of math problem, compared to overall accuracy.

Citation

@article{hendrycksmath2021,
  title={Measuring Mathematical Problem Solving With the MATH Dataset},
  author={Dan Hendrycks
    and Collin Burns
    and Saurav Kadavath
    and Akul Arora
    and Steven Basart
    and Eric Tang
    and Dawn Song
    and Jacob Steinhardt},
  journal={arXiv preprint arXiv:2103.03874},
  year={2021}
}

Spaces:

evaluate-metric
/

competition_math

Runtime error