|
import streamlit as st |
|
from streamlit_option_menu import option_menu |
|
import pandas as pd |
|
from datetime import datetime |
|
import pytz |
|
import time |
|
|
|
|
|
st.set_page_config(page_title="AEOLLM", page_icon="👋", layout="wide") |
|
st.title("NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) Task") |
|
|
|
|
|
with st.sidebar: |
|
page = option_menu( |
|
"Navigation", |
|
["LeaderBoard", "Introduction", "Methodology", "Datasets", "Important Dates", |
|
"Evaluation Metrics", "Submit", "Organisers", "References"], |
|
icons=['trophy', 'house', 'book', 'database', 'calendar', 'clipboard', 'upload', 'people', 'book'], |
|
menu_icon="cast", |
|
default_index=0, |
|
styles={ |
|
"container": {"padding": "5px"}, |
|
"icon": {"color": "orange", "font-size": "18px"}, |
|
"nav-link": {"font-size": "16px", "text-align": "left", "margin":"0px", "--hover-color": "#6c757d"}, |
|
"nav-link-selected": {"background-color": "#FF6347"}, |
|
} |
|
) |
|
|
|
st.markdown(""" |
|
<style> |
|
.dataframe th { |
|
min-width: 100px; |
|
} |
|
.dataframe td { |
|
min-width: 100px; |
|
} |
|
/* 应用到所有的Markdown渲染文本 */ |
|
div[data-testid="stMarkdownContainer"] p, |
|
div[data-testid="stMarkdownContainer"] table, |
|
div[data-testid="stMarkdownContainer"] li { |
|
font-size: 24px; |
|
font-family: 'Times New Roman', serif; |
|
line-height: 1.6; |
|
} |
|
.main-text { |
|
font-size: 24px; |
|
font-family: 'Times New Roman', serif; |
|
line-height: 1.6; |
|
} |
|
</style> |
|
""", unsafe_allow_html=True) |
|
|
|
|
|
if page == "Introduction": |
|
st.header("Introduction") |
|
st.markdown(""" |
|
<p class='main-text'>The Automatic Evaluation of LLMs (AEOLLM) task is a new core task in <a href="http://research.nii.ac.jp/ntcir/ntcir-18">NTCIR-18</a> to support in-depth research on large language models (LLMs) evaluation. |
|
<br />🔍 As LLMs grow popular in both fields of academia and industry, how to effectively evaluate the capacity of LLMs becomes an increasingly critical but still challenging issue. |
|
<br />⚖️ Existing methods can be divided into two types: manual evaluation, which is expensive, and automatic evaluation, which faces many limitations including the task format (the majority belong to multiple-choice questions) and evaluation criteria (occupied by reference-based metrics). |
|
<br />💡 To advance the innovation of automatic evaluation, we proposed the Automatic Evaluation of LLMs (AEOLLM) task which focuses on generative tasks and encourages reference-free methods. Besides, we set up diverse subtasks such as summary generation, non-factoid question answering, text expansion, and dialogue generation to comprehensively test different methods. |
|
<br />🚀 We believe that the AEOLLM task will facilitate the development of the LLMs community.</p> |
|
""", unsafe_allow_html=True) |
|
|
|
elif page == "Methodology": |
|
st.header("Methodology") |
|
col1, col2, col3 = st.columns([1, 3, 1]) |
|
with col2: |
|
st.image("asserts/method.svg", use_column_width=True) |
|
st.markdown(""" |
|
<p class='main-text'>First, we choose four subtasks as shown in the table below:</p> |
|
<table class='main-text'> |
|
<thead> |
|
<tr> |
|
<th style="text-align: left">Task</th> |
|
<th style="text-align: left">Description</th> |
|
<th style="text-align: left">Dataset</th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<td style="text-align: left">Summary Generation (SG)</td> |
|
<td style="text-align: left">write a summary for the specified text</td> |
|
<td style="text-align: left">XSum: over 226k news articles</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: left">Non-Factoid QA (NFQA)</td> |
|
<td style="text-align: left">construct long-form answers to open-ended non-factoid questions</td> |
|
<td style="text-align: left">NF_CATS: 12k non-factoid questions</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: left">Text Expansion (TE)</td> |
|
<td style="text-align: left">given a theme, participants need to generate stories related to the theme</td> |
|
<td style="text-align: left">WritingPrompts: 303k story themes</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: left">Dialogue Generation (DG)</td> |
|
<td style="text-align: left">generate human-like responses to numerous topics in daily conversation contexts</td> |
|
<td style="text-align: left">DailyDialog: 13k daily conversation contexts</td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
<p class='main-text'>Second, we choose a series of popular LLMs during the competition to generate answers.</p> |
|
<p class='main-text'>Third, we manually annotate the answer sets for each question, which will be used as gold standards for evaluating the performance of different evaluation methods.</p> |
|
<p class='main-text'>Last, we will collect evaluation results from participants and calculate consistency with manually annotated results. We will use Accuracy, Kendall’s tau and Spearman correlation coefficient as the evaluation metrics.</p> |
|
""",unsafe_allow_html=True) |
|
|
|
elif page == "Datasets": |
|
st.header("Answer Generation") |
|
st.markdown(""" |
|
We randomly sampled **100 instances** from **each** dataset as the question set and selected **7 different LLMs** to generate answers, forming the answer set. |
|
|
|
As a result, each dataset produced 700 instances, totaling **2,800 instances across the four datasets**. |
|
""") |
|
st.header("Human Annotation") |
|
st.markdown(""" |
|
- For each instance (question-answer pair), we employed human annotators to provide a score ranging from 1 to 5 and took the median of these scores as the final score. |
|
|
|
- Based on this score, we calculated the rankings of the 7 answers for each question. If scores were identical, the answers were assigned the same rank, with the lowest rank being used. |
|
""") |
|
st.header("Data Acquisition and Usage") |
|
st.markdown(""" |
|
We divided the 2,800 instances into three parts: |
|
|
|
1️⃣ train set: 20% of the data (covering all four datasets) was designated as the training set (including human annotations) for participants to reference when designing their methods. |
|
|
|
2️⃣ test set: Another 20% of the data was set aside as the test set (excluding human annotations), used to evaluate the performance of participants' methods and to generate the **leaderboard**. |
|
|
|
3️⃣ reserved set: The remaining 60% of the data was reserved for **the final evaluation**. |
|
|
|
Both the training set and the test set can be downloaded from the provided link: [https://huggingface.co/datasets/THUIR/AEOLLM](https://huggingface.co/datasets/THUIR/AEOLLM). |
|
""") |
|
st.header("Resources") |
|
st.markdown(""" |
|
<p class='main-text'>A brief description of the specific dataset we used, along with the original download link, is provided below:</p> |
|
<p class='main-text'>1. <strong>Summary Generation (SG): <a href="https://huggingface.co/datasets/EdinburghNLP/xsum">Xsum</a></strong>: A real-world single document news summary dataset collected from online articles by the British Broadcasting Corporation (BBC) and contains over 220 thousand news documents.</p> |
|
<p class='main-text'>2. <strong>Non-Factoid QA (NFQA): <a href="https://github.com/Lurunchik/NF-CATS">NF_CATS</a></strong>: A dataset contains examples of 12k natural questions divided into eight categories.</p> |
|
<p class='main-text'>3. <strong>Text Expansion (TE): <a href="https://huggingface.co/datasets/euclaise/writingprompts">WritingPrompts</a></strong>: A large dataset of 300K human-written stories paired with writing prompts from an online forum.</p> |
|
<p class='main-text'>4. <strong>Dialogue Generation (DG): <a href="https://huggingface.co/datasets/daily_dialog">DailyDialog</a></strong>: A high-quality dataset of 13k multi-turn dialogues. The language is human-written and less noisy.</p> |
|
""",unsafe_allow_html=True) |
|
|
|
elif page == "Important Dates": |
|
st.header("Important Dates") |
|
st.markdown(""" |
|
<p class='main-text'>All deadlines are at 11:59pm in the Anywhere on Earth (AOE) timezone.</p> |
|
""", unsafe_allow_html=True) |
|
|
|
col1, col2 = st.columns(2) |
|
with col1: |
|
st.markdown(""" |
|
<span class='main-text'><strong>Kickoff Event</strong>:</span> <br /> |
|
<span class='main-text'><strong>Dataset Release</strong>:</span> <br /> |
|
<span class='main-text'><strong>Dry run Deadline</strong>:</span><br /> |
|
<span class='main-text'><strong>Formal run</strong>:</span> <br /> |
|
<span class='main-text'><strong>Evaluation Results Release</strong>:</span> <br /> |
|
<span class='main-text'><strong>Task overview release (draft)</strong>:</span> <br /> |
|
<span class='main-text'><strong>Submission Due of Participant Papers (draft)</strong>:</span> <br /> |
|
<span class='main-text'><strong>Camera-Ready Participant Paper Due</strong>:</span><br /> |
|
<span class='main-text'><strong>NTCIR-18 Conference</strong>:</span> <br /> |
|
""",unsafe_allow_html=True) |
|
with col2: |
|
st.markdown(""" |
|
<span class='main-text'>March 29, 2024</span><br /> |
|
<span class='main-text'>May 1, 2024</span><br /> |
|
<span class='main-text'>👉Jan 15, 2025</span><br /> |
|
<span class='main-text'>Jan 15, 2025 - Feb 1, 2025</span> <br /> |
|
<span class='main-text'>Feb 1, 2025</span> <br /> |
|
<span class='main-text'>Feb 1, 2025</span><br /> |
|
<span class='main-text'>March 1, 2025</span><br /> |
|
<span class='main-text'>May 1, 2025</span><br /> |
|
<span class='main-text'>Jun 10-13 2025</span><br /> |
|
""",unsafe_allow_html=True) |
|
st.markdown(""" |
|
<p>During the Dry run (until Jan 15, 2025), we will use the <a href="https://huggingface.co/datasets/THUIR/AEOLLM">test set (https://huggingface.co/datasets/THUIR/AEOLLM)</a> to evaluate the performance of participants' methods and release the results on the Leaderboard. |
|
<br /> |
|
Before the Formal run begins (before Jan 15, 2025), we will release the reserved set. Participants need to submit their results for the reserved set before the Formal run ends (before Feb 1, 2025).</p> |
|
""",unsafe_allow_html=True) |
|
elif page == "Evaluation Metrics": |
|
st.header("Evaluation Metrics") |
|
st.markdown(""" |
|
- **Acc(Accuracy):** The proportion of identical preference results between the model and human annotations. Specifically, we first convert individual scores (ranks) into pairwise preferences and then calculate consistency with human annotations. |
|
|
|
- **Kendall's tau:** Measures the ordinal association between two ranked variables. |
|
|
|
$$ |
|
\\tau=\\frac{C-D}{\\frac{1}{2}n(n-1)} |
|
$$ |
|
|
|
where: |
|
- $C$ is the number of concordant pairs, |
|
- $D$ is the number of discordant pairs, |
|
- $n$ is the number of pairs. |
|
|
|
- **Spearman's Rank Correlation Coefficient:** Measures the strength and direction of the association between two ranked variables. |
|
|
|
$$ |
|
\\rho = 1 - \\frac{6 \sum d_i^2}{n(n^2 - 1)} |
|
$$ |
|
|
|
where: |
|
- $d_i$ is the difference between the ranks of corresponding elements in the two lists, |
|
- $n$ is the number of elements. |
|
""",unsafe_allow_html=True) |
|
elif page == "Data and File format": |
|
st.header("Data and File format") |
|
st.markdown(""" |
|
<p class='main-text'>We will be following a similar format as the ones used by most <strong>TREC submissions</strong>, which is repeated below. White space is used to separate columns. The width of the columns in the format is not important, but it is important to have exactly five columns per line with at least one space between the columns.</p> |
|
<p class='main-text'><strong>taskId questionId answerId score rank</strong></p> |
|
<p class='main-text'>1. the first column is the taskeId (index different tasks)</p> |
|
<p class='main-text'>2. the second column is questionId (index different questions in the same task)</p> |
|
<p class='main-text'>3. the third column is answerId (index the answer provided by different LLMs to the same question)</p> |
|
<p class='main-text'>4. the fourth column is score (index the score to the answer given by participants)</p> |
|
<p class='main-text'>5. the fifth column is rank (index the rank of the answer within all answers to the same question)</p> |
|
""",unsafe_allow_html=True) |
|
elif page == "Submit": |
|
st.header("File Format") |
|
st.markdown(""" |
|
We will be following a similar format as the ones used by most **TREC submissions**: |
|
|
|
1. White space is used to separate columns. |
|
|
|
2. The width of the columns in the format is not important, but it is important to have exactly five columns per line with at least one space between the columns. |
|
|
|
**taskId questionId answerId score rank** |
|
|
|
- the first column is the taskeId (index different tasks) |
|
- the second column is questionId (index different questions in the same task) |
|
- the third column is answerId (index the answer provided by different LLMs to the same question) |
|
- the fourth column is score (index the score to the answer given by participants) |
|
- the fifth column is rank (index the rank of the answer within all answers to the same question) |
|
""") |
|
st.header("Submit") |
|
st.markdown(""" |
|
📄 Please organize the answers in a **txt** file, name the file as **teamId_methods.txt** and submit it through the link below: [https://forms.gle/vRNxBaNAfYZHMVtr5](https://forms.gle/vRNxBaNAfYZHMVtr5) |
|
|
|
⏱️ Each team can submit up to 5 times per day, and only the latest submission will be considered. |
|
|
|
🔗 An example of the submission file content is [here](https://huggingface.co/spaces/THUIR/AEOLLM/blob/main/baseline_example/output/baseline1_chatglm3_6B.txt). |
|
""") |
|
elif page == "LeaderBoard": |
|
|
|
st.markdown(""" |
|
<p class='main-text'> |
|
🏆 NTCIR-18 Automatic Evaluation Methods of LLMs (AEOLLM) task Leaderboard. |
|
</p> |
|
""", unsafe_allow_html=True) |
|
df = { |
|
"TeamId": ["baseline", "baseline", "baseline", "baseline"], |
|
"Methods": ["chatglm3-6b", "baichuan2-13b", "chatglm-pro", "gpt-4o"], |
|
"Average (all 4 datatsets)": [], |
|
"Average (Dialogue Generation)": [], |
|
"Accuracy (Dialogue Generation)": [], |
|
"Kendall's Tau (Dialogue Generation)": [], |
|
"Spearman (Dialogue Generation)": [], |
|
"Average (Text Expansion)": [], |
|
"Accuracy (Text Expansion)": [], |
|
"Kendall's Tau (Text Expansion)": [], |
|
"Spearman (Text Expansion)": [], |
|
"Average (Summary Generation)": [], |
|
"Accuracy (Summary Generation)": [], |
|
"Kendall's Tau (Summary Generation)": [], |
|
"Spearman (Summary Generation)": [], |
|
"Average (Non-Factoid QA)": [], |
|
"Accuracy (Non-Factoid QA)": [], |
|
"Kendall's Tau (Non-Factoid QA)": [], |
|
"Spearman (Non-Factoid QA)": [], |
|
} |
|
|
|
TeamId = ["baseline", "baseline", "baseline", "baseline", 'ISLab', 'ISLab', 'ISLab', 'ISLab', 'default5', 'default5'] |
|
Methods = ["chatglm3-6b", "baichuan2-13b", "chatglm-pro", "gpt-4o", "llama3-1_baseline5", "llama3-1_baseline6", "llama3-1-baseline7", "llama3-2-baseline", "llm", "baselinev00"] |
|
|
|
|
|
DG = { |
|
"TeamId": TeamId, |
|
"Methods": Methods, |
|
"Accuracy": [0.5806, 0.5483, 0.6001, 0.6472, 0, 0, 0, 0, 0.6504481792717087, 0.5816503267973856], |
|
"Kendall's Tau": [0.3243, 0.1739, 0.3042, 0.4167, 0, 0, 0, 0, 0.4034134076281578, 0.25873621367415467], |
|
"Spearman": [0.3505, 0.1857, 0.3264, 0.4512, 0, 0, 0, 0, 0.4303514807222638, 0.28208851475394925], |
|
} |
|
df1 = pd.DataFrame(DG) |
|
|
|
TE = { |
|
"TeamId": TeamId, |
|
"Methods": Methods, |
|
"Accuracy": [0.5107, 0.5050, 0.5461, 0.5581, 0.5067545088210725, 0.4766805549971185, 0, 0, 0.511188817632316, 0.5369175431336809], |
|
"Kendall's Tau": [0.1281, 0.0635, 0.2716, 0.3864, 0.18884532500063825, 0.31629653258509166, 0, 0, 0.10828008098753536, 0.26854542496891], |
|
"Spearman": [0.1352, 0.0667, 0.2867, 0.4157, 0.2033137543983765, 0.35189638758373964, 0, 0, 0.11421806788123415, 0.2820788649343955], |
|
} |
|
df2 = pd.DataFrame(TE) |
|
|
|
SG = { |
|
"TeamId": TeamId, |
|
"Methods": Methods, |
|
"Accuracy": [0.6504, 0.6014, 0.7162, 0.7441, 0.7518953983108395, 0.7870818213649097, 0.6187623875307698, 0.8003185213479332, 0.7007955341227401, 0.7519048414820473], |
|
"Kendall's Tau": [0.3957, 0.2688, 0.5092, 0.5001, 0.5377072309689559, 0.5709963447418871, 0.30897221697376714, 0.6064826537169805, 0.4819411311747811, 0.4874144871543796], |
|
"Spearman": [0.4188, 0.2817, 0.5403, 0.5405, 0.5830423197486431, 0.6276373633425562, 0.324348752437819, 0.6664032039425867, 0.5076789062134682, 0.539580429716673], |
|
} |
|
df3 = pd.DataFrame(SG) |
|
|
|
NFQA = { |
|
"TeamId": TeamId, |
|
"Methods": Methods, |
|
"Accuracy": [0.5935, 0.5817, 0.7000, 0.7203, 0, 0, 0, 0, 0.5922294372294372, 0.7146378968253966], |
|
"Kendall's Tau": [0.2332, 0.2389, 0.4440, 0.4235, 0, 0, 0, 0, 0.1701874070113157, 0.44019963513470517], |
|
"Spearman": [0.2443, 0.2492, 0.4630, 0.4511, 0, 0, 0, 0, 0.18058287732894646, 0.4592548426675891], |
|
} |
|
df4 = pd.DataFrame(NFQA) |
|
|
|
OverAll = { |
|
"TeamId": TeamId, |
|
"Methods": Methods, |
|
"Accuracy": [], |
|
"Kendall's Tau": [], |
|
"Spearman": [] |
|
} |
|
|
|
data = [DG, NFQA, SG, TE] |
|
task = ["Dialogue Generation", "Non-Factoid QA", "Summary Generation", "Text Expansion"] |
|
metric = ["Accuracy", "Kendall's Tau", "Spearman"] |
|
|
|
for m in metric: |
|
|
|
metric_score = [0] * len(TeamId) |
|
for j in range(len(TeamId)): |
|
|
|
for d in data: |
|
metric_score[j] += d[m][j] |
|
metric_score = [k / len(task) for k in metric_score] |
|
OverAll[m] = metric_score |
|
|
|
dfo = pd.DataFrame(OverAll) |
|
|
|
df = [df1, df2, df3, df4, dfo] |
|
for d in df: |
|
for col in d.select_dtypes(include=['float64', 'int64']).columns: |
|
d[col] = d[col].apply(lambda x: f"{x:.4f}") |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
st.markdown("""<p class='main-text'>Overall: The average across all four tasks</p>""", unsafe_allow_html=True) |
|
st.dataframe(dfo, use_container_width=True) |
|
|
|
st.markdown("""<p class='main-text'>Task: Dialogue Generation; Dataset: DialyDialog</p>""", unsafe_allow_html=True) |
|
st.dataframe(df1, use_container_width=True) |
|
|
|
st.markdown("""<p class='main-text'>Task: Text Expansion; Dataset: WritingPrompts</p>""", unsafe_allow_html=True) |
|
st.dataframe(df2, use_container_width=True) |
|
|
|
st.markdown("""<p class='main-text'>Task: Summary Generation; Dataset: Xsum</p>""", unsafe_allow_html=True) |
|
st.dataframe(df3, use_container_width=True) |
|
|
|
st.markdown("""<p class='main-text'>Task: Non-Factoid QA; Dataset: NF_CATS</p>""", unsafe_allow_html=True) |
|
st.dataframe(df4, use_container_width=True) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
st.markdown(""" |
|
🔗 To register for AEOLLM task, you can visit the following link and choose our AEOLLM task: [https://research.nii.ac.jp/ntcir/ntcir-18/howto.html](https://research.nii.ac.jp/ntcir/ntcir-18/howto.html). |
|
|
|
📃 To submit, refer to the "Submit" section in the left-hand navigation bar.🤗 A baseline example can be found in the [baseline_example](https://huggingface.co/spaces/THUIR/AEOLLM/tree/main/baseline_example) folder. |
|
|
|
📝 Refer to other sections in the navigation bar for details on evaluation metrics, datasets, important dates and methodology. |
|
|
|
🕒 The Leaderboard will be updated daily around 24:00 Beijing Time. |
|
""") |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
elif page == "Organisers": |
|
st.header("Organisers") |
|
st.markdown(""" |
|
<p class='main-text'> |
|
<em>Yiqun Liu</em> [yiqunliu@tsinghua.edu.cn] (Tsinghua University)<br /> |
|
<em>Qingyao Ai</em> [aiqy@tsinghua.edu.cn] (Tsinghua University)<br /> |
|
<em>Junjie Chen</em> [chenjj826@gmail.com] (Tsinghua University) <br /> |
|
<em>Zhumin Chu</em> [chuzm19@mails.tsinghua.edu.cn] (Tsinghua University)<br /> |
|
<em>Haitao Li</em> [liht22@mails.tsinghua.edu.cn] (Tsinghua University)<br /> |
|
Please feel free to contact us! 😉 |
|
</p>""",unsafe_allow_html=True) |
|
st.image("asserts/organizer.png") |
|
elif page == "References": |
|
st.header("References") |
|
st.markdown(""" |
|
<p class='main-text'>[1] Mao R, Chen G, Zhang X, et al. GPTEval: A survey on assessments of ChatGPT and GPT-4. <a href="https://arxiv.org/pdf/2308.12488">pdf</a><br /> |
|
[2] Chang Y, Wang X, Wang J, et al. A survey on evaluation of large language models. <a href="https://dl.acm.org/doi/pdf/10.1145/3641289">pdf</a><br /> |
|
[3] Chan C M, Chen W, Su Y, et al. Chateval: Towards better llm-based evaluators through multi-agent debate. <a href="https://arxiv.org/pdf/2308.07201">pdf</a><br /> |
|
[4] Li R, Patel T, Du X. Prd: Peer rank and discussion improve large language model based evaluations. <a href="https://arxiv.org/pdf/2307.02762">pdf</a><br /> |
|
[5] Chu Z, Ai Q, Tu Y, et al. Pre: A peer review based large language model evaluator. <a href="https://arxiv.org/pdf/2401.15641">pdf</a></p> |
|
""",unsafe_allow_html=True) |
|
|
|
|
