Merge remote-tracking branch 'gh-origin/hf'

README.md

@@ -1,5 +1,5 @@
 ---
-title: LLMRiddles
+title: LLMRiddles-ChatGPT-CN
 emoji: 🚀
 colorFrom: indigo
 colorTo: green
@@ -21,11 +21,19 @@ python_version: 3.8
 	<br>
 </div>
 
+English | [简体中文](https://github.com/opendilab/LLMRiddles/blob/main/README_zh.md)
+
 ## :thinking: What's This
 Welcome to LLM Riddles! This is a game of wits and courage with language models. In the game, you need to construct questions that interact with the language model to get answers that meet the requirements. In this process, you can use your brain and use all the methods you can think of to get the model to output the results required by the answer.
 
 ## :space_invader: How to Play
-We provide an online version for players to directly access and try out. Local deployment can be done in the following ways:
+We provide an online version for players to directly access and try out. 
+- [ChatGPT + English(w/o key)](https://huggingface.co/spaces/OpenDILabCommunity/LLMRiddlesChatGPTEN)
+- [ChatGPT + Chinese(w/o key)](https://huggingface.co/spaces/OpenDILabCommunity/LLMRiddlesChatGPTCN)
+- [Mistral + English(w/ key)](https://4521e4d138d3779498.gradio.live)
+- [ChatGPT + Chinese(w/ key)](http://llmriddles.opendilab.net/)
+
+Local deployment can be done in the following ways:
 ### ChatGPT + Chinese
 ```shell
 QUESTION_LANG=cn QUESTION_LLM='chatgpt' QUESTION_LLM_KEY=<your API key> python3 -u app.py
@@ -50,14 +58,15 @@ The question format should include the following points:
 - Modify the corresponding chapter question files
 - Modification of init.py
 
-For a complete example, please refer to: [Submit your own level design]()
+For a complete example, please refer to: [Submit your own level design](https://github.com/opendilab/LLMRiddles/pull/6)
 
 ## :writing_hand: Roadmap
 
 - [x] Support custom levels
-- [ ] Online trial link
-- [ ] Hugging Face Space link
+- [x] Online trial link
+- [x] Hugging Face Space link
 - [x] Support Mistral-7B（English version）
+- [ ] Support ChatGLM（Chinese version）
 - [ ] Support Baichuan2-7B（Chinese version）
 - [ ] Support LLaMA2-7B（English version）
 - [ ] LLM inference speed optimization
@@ -68,12 +77,12 @@ For a complete example, please refer to: [Submit your own level design]()
 - Discuss on OpenDILab's WeChat group (i.e. add us on WeChat: ding314assist)
 <img src=https://github.com/opendilab/LLMRiddles/blob/main/llmriddles/assets/wechat.jpeg width=35% />
 
-## Special Thanks
+## :star2: Special Thanks
 - Thanks to [Haoqiang Fan](https://www.zhihu.com/people/haoqiang-fan) for his original idea and title, which provided inspiration and motivation for the development and expansion of this project.
 - Thanks to [HuggingFace](https://huggingface.co) for supporting and assisting the game.
 - Thanks to [LLM Riddles contributors](https://github.com/opendilab/LLMRiddles/graphs/contributors) for their implementation and support.
 
-## License
+## :label: License
 All code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
 
 <p align="right">(<a href="#top">back to top</a>)</p>

README_zh.md

@@ -8,11 +8,19 @@
 	<br>
 </div>
 
+[English](https://github.com/opendilab/LLMRiddles/blob/main/README.md) | 简体中文
+
 ## :thinking: 什么是LLM Riddles
 欢迎来到 LLM Riddles！这是一个与语言模型斗智斗勇的游戏。在游戏中，你需要构造与语言模型交互的问题，来得到符合要求的答案。在这个过程中，你可以开动脑筋，用你想到的所有方式，让模型输出答案要求的结果。
 
 ## :space_invader: 如何试玩
-我们提供了在线版本以供玩家直接访问试玩，本地部署可以通过以下方式：
+我们提供了在线版本以供玩家直接访问试玩:
+- [ChatGPT + 英文(需配置api key)](https://huggingface.co/spaces/OpenDILabCommunity/LLMRiddlesChatGPTEN)
+- [ChatGPT + 中文(需配置api key)](https://huggingface.co/spaces/OpenDILabCommunity/LLMRiddlesChatGPTCN)
+- [Mistral + 英文(已预设api key)](https://4521e4d138d3779498.gradio.live)
+- [ChatGPT + 中文(已预设api key)](http://llmriddles.opendilab.net/)
+
+本地部署可以通过以下方式：
 ### ChatGPT + 中文
 ```shell
 QUESTION_LANG=cn QUESTION_LLM='chatgpt' QUESTION_LLM_KEY=<your API key> python3 -u app.py
@@ -42,16 +50,17 @@ QUESTION_LANG=en QUESTION_LLM='llama2-7b' python3 -u app.py
 - 对应章节问题文件的修改
 - init.py的修改
 
-完整示例请参考：[提交属于自己的关卡设计]()
+完整示例请参考：[提交属于自己的关卡设计](https://github.com/opendilab/LLMRiddles/pull/6)
 
 ## :writing_hand: 未来计划
 
 - [x] 支持自定义关卡
-- [ ] 在线试玩链接
-- [ ] Hugging Face Space 链接
-- [ ] 支持LLaMA2-7B（英文）
-- [ ] 支持Mistral-7B（英文）
+- [x] 在线试玩链接
+- [x] Hugging Face Space 链接
+- [x] 支持Mistral-7B（英文）
+- [ ] 支持ChatGLM（中文）
 - [ ] 支持Baichuan2-7B（中文）
+- [ ] 支持LLaMA2-7B（英文）
 - [ ] LLM 推理速度优化
 
 ## :speech_balloon: 反馈问题 & 提出建议
@@ -60,12 +69,12 @@ QUESTION_LANG=en QUESTION_LLM='llama2-7b' python3 -u app.py
 - 在OpenDILab的群组中加入讨论(通过 WeChat: ding314assist 添加小助手微信)
 <img src=https://github.com/opendilab/LLMRiddles/blob/main/llmriddles/assets/wechat.jpeg width=35% />
 
-## Special Thanks
+## :star2: Special Thanks
 - 感谢 [Haoqiang Fan](https://www.zhihu.com/people/haoqiang-fan) 的原始创意和题目，为本项目的开发和扩展提供了灵感与动力。
 - 感谢 [HuggingFace](https://huggingface.co) 对游戏的支持与协助。
 - 感谢 [LLM Riddles contributors](https://github.com/opendilab/LLMRiddles/graphs/contributors) 的实现与支持。
 
-## License
+## :label: License
 All code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
 
 <p align="right">(<a href="#top">back to top</a>)</p>

app.py

@@ -1,13 +1,13 @@
+import logging
 import os
 import uuid
-import logging
 
 import gradio as gr
 
 from llmriddles.questions import QuestionExecutor
 from llmriddles.questions import list_ordered_questions
 
-_QUESTION_IDS = {}
+_QUESTION_SESSIONS = {}
 count = 0
 _QUESTIONS = list_ordered_questions()
 _LANG = os.environ.get('QUESTION_LANG', 'cn')
@@ -17,21 +17,18 @@ assert _LLM in ['chatgpt', 'mistral-7b'], _LLM
 _LLM_KEY = os.environ.get('QUESTION_LLM_KEY', None)
 _DEBUG = os.environ.get('DEBUG', 'false').lower() == 'true'
 
+if _DEBUG:
+    logging.getLogger().setLevel(logging.INFO)
+else:
+    logging.getLogger().setLevel(logging.WARNING)
 if _LANG == "cn":
-    if _DEBUG:
-        logging.getLogger().setLevel(logging.INFO)
-    else:
-        logging.getLogger().setLevel(logging.WARNING)
     title = "完蛋！我被 LLM 拿捏了"
     requirement_ph = """
-    欢迎来到 LLM Riddles!
-
-    你将通过本游戏对大语言模型产生更深刻的理解。在本游戏中，你需要构造一个提给语言大模型的问题，使得它回复的答案符合题目要求。
-
-    点击\"下一题\"即可开始游戏
+    <h2 style="color: #6d28d9;"> 欢迎来到 LLM Riddles! </h2>
+    <h4> 你将通过本游戏对大语言模型产生更深刻的理解。在本游戏中，你需要构造一个提给语言大模型的问题，使得它回复的答案符合题目要求。点击<i>\"下一题\"</i> 即可开始游戏。</h4>
     """
     requirement_label = "游戏须知/说明"
-    question_ph = "你对大语言模型的提问"
+    question_ph = "你对大语言模型的提问（例如：请你输出1+1=3）"
     question_label = "玩家提问栏"
     answer_ph = "大语言模型的回答"
     answer_label = "大语言模型回答栏"
@@ -41,17 +38,19 @@ if _LANG == "cn":
     api_label = "API key"
     predict_label = "结果正确性"
     explanation_label = "结果详细解释"
-    game_cleared_label = "祝贺！你已成功通关！"
+    game_cleared_label = "<h2 style='color: #6d28d9;'>祝贺！你已成功通关！</h2>"
     correct_label = "正确"
     wrong_label = "错误"
     api_error_info = "请在提交问题之前先输入你的 API Key"
     try_again_label = "再玩一次"
+    select_label = "选择关卡（投机取巧需谨慎）"
     title_markdown = """
     <div align="center">
         <img src="https://raw.githubusercontent.com/opendilab/LLMRiddles/main/llmriddles/assets/banner.svg" width="80%" height="20%" alt="Banner Image">
     </div>
-    <h2 style="text-align: center; color: black;"><a href="https://github.com/OpenDILab"> 🎭LLM Riddles：完蛋！我被 LLM 拿捏了</a></h2>
-    <h4 align="center"> 如果你喜欢这个项目，请给我们在 GitHub 点个 star ✨ 。我们将会持续保持更新。再次感谢游戏<a href="https://www.zhihu.com/people/haoqiang-fan"> 原作者 </a>的奇思妙想！  </h4>
+    <h2 style="text-align: center; color: black;"><a href="https://github.com/OpenDILab/LLMRiddles"> 🎭LLM Riddles：完蛋！我被 LLM 拿捏了</a></h2>
+    <strong><h5 align="center"> 更多不同语言模型的在线试玩 demo 可以访问 GitHub<a href="https://github.com/OpenDILab/LLMRiddles">源代码仓库</a>获取<h5></strong>
+    <h5 align="center"> 如果你喜欢这个项目，请给我们在 GitHub 点个 star ✨ <a href="https://github.com/OpenDILab/LLMRiddles"> 代码仓库传送门 </a> 。我们将会持续保持更新。再次感谢游戏<a href="https://www.zhihu.com/people/haoqiang-fan"> 原作者 </a>的奇思妙想！  </h5>
     <strong><h5 align="center">注意：算法模型的输出可能包含一定的随机性。相关结果不代表任何开发者和相关 AI 服务的态度和意见。本项目开发者不对生成结果作任何保证，仅供娱乐。<h5></strong>
     """
     tos_markdown = """
@@ -65,14 +64,11 @@ if _LANG == "cn":
 elif _LANG == "en":
     title = "LLM Riddles: Oops! Rolling in LLM."
     requirement_ph = """
-    Welcome to LLM Riddles!
-
-    In this game, you'll gain a deeper understanding of language models. Your challenge is to create a question to ask a language model in a way that the answer it provides meets specific criteria.
-
-    Click \'Next\' to Start
+    <h2 style="color: #6d28d9;">Welcome to LLM Riddles! </h2>
+    <h4> In this game, you'll gain a deeper understanding of language models. Your challenge is to create a question to ask a language model in a way that the answer it provides meets specific criteria. Click <i>\'Next\'</i> to Start</h4>
     """
     requirement_label = "Game Requirements"
-    question_ph = "Your Question for LLM"
+    question_ph = "Your Question for LLM (e.g. Please print 1+1=3)"
     question_label = "Question"
     answer_ph = "Answer From LLM"
     answer_label = "Answer"
@@ -82,17 +78,18 @@ elif _LANG == "en":
     api_label = "API key"
     predict_label = "Correctness"
     explanation_label = "Explanation"
-    game_cleared_label = "Congratulations!"
+    game_cleared_label = "<h2 style='color: #6d28d9;'>Congratulations!</h2>"
     correct_label = "Correct"
     wrong_label = "Wrong"
     api_error_info = "Please Enter API Key Before Submitting Question."
     try_again_label = "Try Again"
+    select_label = "Select level"
     title_markdown = """
     <div align="center">
         <img src="https://raw.githubusercontent.com/opendilab/LLMRiddles/main/llmriddles/assets/banner.svg" width="80%" height="20%" alt="Banner Image">
     </div>
-    <h2 style="text-align: center; color: black;"><a href="https://github.com/OpenDILab"> 🎭LLM Riddles: Oops! Rolling in LLM.</a></h2>
-    <h4 align="center"> If you like our project, please give us a star ✨ on GitHub for latest update. Thanks for the interesting idea of the original game <a href="https://www.zhihu.com/people/haoqiang-fan"> author </a>.  </h4>
+    <h2 style="text-align: center; color: black;"><a href="https://github.com/OpenDILab/LLMRiddles"> 🎭LLM Riddles: Oops! Rolling in LLM.</a></h2>
+    <h5 align="center"> If you like our project, please give us a star ✨ on GitHub for latest update <a href="https://github.com/OpenDILab/LLMRiddles"> (Code Link) </a>. Thanks for the interesting idea of the original game <a href="https://www.zhihu.com/people/haoqiang-fan"> author </a>.  </h5>
     <strong><h5 align="center">Notice: The output is generated by algorithm scheme and may involve some randomness. It does not represent the attitudes and opinions of any developers and AI services in this project. We do not make any guarantees about the generated content.<h5></strong>
     """
     tos_markdown = """
@@ -123,7 +120,7 @@ if __name__ == '__main__':
         gr.Markdown(title_markdown)
 
         with gr.Row():
-            gr_requirement = gr.TextArea(placeholder=requirement_ph, label=requirement_label, lines=4)
+            gr_requirement = gr.HTML(value=requirement_ph, label=requirement_label)
         with gr.Row():
             with gr.Column():
                 gr_question = gr.TextArea(placeholder=question_ph, label=question_label)
@@ -131,6 +128,11 @@ if __name__ == '__main__':
                 with gr.Row():
                     gr_submit = gr.Button(submit_label, interactive=False)
                     gr_next = gr.Button(next_label)
+                with gr.Row():
+                    gr_select = gr.Radio(
+                        choices=[(QuestionExecutor(q, _LANG).question_name, i) for i, q in enumerate(_QUESTIONS)],
+                        label=select_label
+                    )
 
             with gr.Column():
                 gr_uuid = gr.Text(value='', visible=False)
@@ -139,6 +141,48 @@ if __name__ == '__main__':
                 gr_explanation = gr.TextArea(label=explanation_label, lines=1)
         gr.Markdown(tos_markdown)
 
+        def _postprocess_question_text(question_text):
+            if _LANG == 'cn':
+                idx = question_text.find('，')
+                question_title = question_text[:idx]
+                former, latter = question_title.split('（')
+                question_title = former + '：' + latter[:-1]
+                question_text = f"<h2 style='color: #6d28d9;'>{question_title}</h2><h4>{question_text[idx+1:]}</h4>"
+            elif _LANG == 'en':
+                idx = question_text.find(',')
+                question_text = f"<h2 style='color: #6d28d9;'>{question_text[:idx]}</h2><h4>{question_text[idx+1:]}</h4>"
+            return question_text
+
+
+        def _radio_select(uuid_, select_qid):
+            global count
+            if not uuid_:
+                uuid_ = str(uuid.uuid4())
+                count += 1
+                logging.info(f'Player {count} starts the game now')
+            global _QUESTION_SESSIONS
+            if uuid_ not in _QUESTION_SESSIONS:
+                _QUESTION_SESSIONS[uuid_] = set(), select_qid
+            else:
+                _exists, _ = _QUESTION_SESSIONS[uuid_]
+                _QUESTION_SESSIONS[uuid_] = _exists, select_qid
+
+            executor = QuestionExecutor(_QUESTIONS[select_qid], _LANG)
+            question_text = _postprocess_question_text(executor.question_text)
+            return question_text, '', '', {}, '', \
+                gr.Button(submit_label, interactive=True), \
+                gr.Button(next_label, interactive=False), \
+                uuid_
+
+        gr_select.select(
+            _radio_select,
+            inputs=[gr_uuid, gr_select],
+            outputs=[
+                gr_requirement, gr_question, gr_answer,
+                gr_predict, gr_explanation, gr_submit, gr_next, gr_uuid,
+            ],
+        )
+
 
         def _next_question(uuid_):
             global count
@@ -146,40 +190,56 @@ if __name__ == '__main__':
                 uuid_ = str(uuid.uuid4())
                 count += 1
                 logging.info(f'Player {count} starts the game now')
-            global _QUESTION_IDS
-            _qid = _QUESTION_IDS.get(uuid_, -1)
+            global _QUESTION_SESSIONS
+            if uuid_ in _QUESTION_SESSIONS:
+                _exists, _qid = _QUESTION_SESSIONS[uuid_]
+            else:
+                _exists, _qid = set(), -1
             _qid += 1
-            _QUESTION_IDS[uuid_] = _qid
+            _QUESTION_SESSIONS[uuid_] = _exists, _qid
 
             if _qid >= len(_QUESTIONS):
-                del _QUESTION_IDS[uuid_]
+                del _QUESTION_SESSIONS[uuid_]
                 logging.info(f'Player {count} has passed the game now')
                 return game_cleared_label, '', '', {}, '', \
                     gr.Button(submit_label, interactive=False), \
                     gr.Button(try_again_label, interactive=True), \
-                    ''
+                    '', \
+                    gr.Radio(
+                        choices=[(QuestionExecutor(q, _LANG).question_name, i) for i, q in enumerate(_QUESTIONS)],
+                        label=select_label
+                    )
             else:
                 executor = QuestionExecutor(_QUESTIONS[_qid], _LANG)
-                return executor.question_text, '', '', {}, '', \
+                question_text = _postprocess_question_text(executor.question_text)
+                return question_text, '', '', {}, '', \
                     gr.Button(submit_label, interactive=True), \
                     gr.Button(next_label, interactive=False), \
-                    uuid_
+                    uuid_, \
+                    gr.Radio(
+                        choices=[(QuestionExecutor(q, _LANG).question_name, i) for i, q in enumerate(_QUESTIONS)],
+                        value=_qid,
+                        label=select_label,
+                    )
+
 
         gr_next.click(
             fn=_next_question,
             inputs=[gr_uuid],
             outputs=[
                 gr_requirement, gr_question, gr_answer,
-                gr_predict, gr_explanation, gr_submit, gr_next, gr_uuid,
+                gr_predict, gr_explanation, gr_submit, gr_next,
+                gr_uuid, gr_select,
             ],
         )
 
 
         def _submit_answer(qs_text: str, api_key: str, uuid_: str):
+            global _QUESTION_SESSIONS
             if _need_api_key() and not api_key:
                 raise gr.Error(api_error_info)
 
-            _qid = _QUESTION_IDS[uuid_]
+            _exists, _qid = _QUESTION_SESSIONS[uuid_]
             executor = QuestionExecutor(
                 _QUESTIONS[_qid], _LANG,
                 llm=_LLM, llm_cfgs=_get_api_key_cfgs(api_key) if _need_api_key() else {'api_key': _LLM_KEY}
@@ -187,10 +247,12 @@ if __name__ == '__main__':
             answer_text, correctness, explanation = executor.check(qs_text)
             labels = {correct_label: 1.0} if correctness else {wrong_label: 1.0}
             if correctness:
+                _QUESTION_SESSIONS[uuid_] = (_exists | {_qid}), _qid
                 return answer_text, labels, explanation, gr.Button(next_label, interactive=True), uuid_
             else:
                 return answer_text, labels, explanation, gr.Button(next_label, interactive=False), uuid_
 
+
         gr_submit.click(
             _submit_answer,
             inputs=[gr_question, gr_api_key, gr_uuid],

llmriddles/questions/executor.py

@@ -15,6 +15,10 @@ class QuestionExecutor:
     def question_text(self):
         return self.question.texts[self.lang]
 
+    @property
+    def question_name(self):
+        return self.question.names[self.lang]
+
     def check(self, qs_text: str) -> Tuple[str, bool, str]:
         answer_text = get_llm_fn(self.llm)(qs_text, **self.llm_cfgs)
         correct, explanation = self.check_answer(qs_text, answer_text)

llmriddles/questions/level1.py

@@ -1,14 +1,18 @@
 from .question import register_question
 
+
 def count_english_words(text: str):
     return len(text.split(' '))
 
+
 def count_chinese_words(text: str):
     return len(text)
 
+
 def check_if_chinese(text: str):
     return all('\u4e00' <= char <= '\u9fff' for char in text)
 
+
 def count_words(text: str, contain_punctuation: bool = False):
     chinese_words = []
     english_words = []
@@ -29,9 +33,9 @@ def count_words(text: str, contain_punctuation: bool = False):
                     temp_english_words = []
                 other_words.append(char)
     if contain_punctuation:
-        return len(chinese_words)+len(english_words)+len(other_words)
+        return len(chinese_words) + len(english_words) + len(other_words)
     else:
-        return len(chinese_words)+len(english_words)
+        return len(chinese_words) + len(english_words)
 
 
 CN_TEXT_1 = """
@@ -57,11 +61,14 @@ def _checker_1(question_text: str, user_text: str, answer_text: str, lang: str):
         return False, None
 
 
-register_question({
-    'cn': CN_TEXT_1,
-    'en': EN_TEXT_1,
-}, _checker_1)
-
+register_question(
+    {
+        'cn': CN_TEXT_1,
+        'en': EN_TEXT_1,
+    },
+    checkers=_checker_1,
+    name={'cn': '1-1 初来乍到', 'en': '1-1'},
+)
 
 CN_TEXT_2 = """
 第一章第二题（小试牛刀），请你输入三个字（及）以内的问题，使模型的回答在30个字以上。
@@ -89,11 +96,14 @@ def _checker_2(question_text: str, user_text: str, answer_text: str, lang: str):
         return True, None
 
 
-register_question({
-    'cn': CN_TEXT_2,
-    'en': EN_TEXT_2,
-}, _checker_2)
-
+register_question(
+    {
+        'cn': CN_TEXT_2,
+        'en': EN_TEXT_2,
+    },
+    checkers=_checker_2,
+    name={'cn': '1-2 小试牛刀', 'en': '1-2'},
+)
 
 CN_TEXT_3 = """
 第一章第三题（短说长话），请你输入一个字的问题，使模型的回答在100个字以上。
@@ -107,6 +117,7 @@ For the third question in chapter 1, please enter a one-word question so that th
 Please enter your query below and click the submit button
 """
 
+
 def _checker_3(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
@@ -119,11 +130,15 @@ def _checker_3(question_text: str, user_text: str, answer_text: str, lang: str):
     else:
         return True, None
 
-register_question({
-    'cn': CN_TEXT_3,
-    'en': EN_TEXT_3,
-}, _checker_3)
 
+register_question(
+    {
+        'cn': CN_TEXT_3,
+        'en': EN_TEXT_3,
+    },
+    checkers=_checker_3,
+    name={'cn': '1-3 短说长话', 'en': '1-3'}
+)
 
 CN_TEXT_4 = """
 第一章第四题（短说短话），请输入一个字的问题，使模型的回答字数小于20个字。
@@ -137,6 +152,7 @@ For the fourth question in chapter 1, please enter a one-word question so that t
 Please enter your query below and click the submit button
 """
 
+
 def _checker_4(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
@@ -145,15 +161,19 @@ def _checker_4(question_text: str, user_text: str, answer_text: str, lang: str):
     if count_words(user_text) > 1:
         return False, "用户的问题长度应该在一个字及以内" if lang == 'cn' else 'Question should be one word.'
     elif count_words(answer_text) >= 20:
-        return False, "大语言模型的答案应该小于100个字" if lang == 'cn' else 'Answer should be less than 20 words.'
+        return False, "大语言模型的答案应该小于20个字" if lang == 'cn' else 'Answer should be less than 20 words.'
     else:
         return True, None
 
-register_question({
-    'cn': CN_TEXT_4,
-    'en': EN_TEXT_4,
-}, _checker_4)
 
+register_question(
+    {
+        'cn': CN_TEXT_4,
+        'en': EN_TEXT_4,
+    },
+    checkers=_checker_4,
+    name={'cn': '1-4 短说短话', 'en': '1-4'},
+)
 
 # CN_TEXT_5 = """
 # 第一章第五题（回文不变），请输入一个本身不是回文串的问题，使无论正着问还是倒着问，模型的回答是一样的。

llmriddles/questions/level2.py

@@ -54,13 +54,18 @@ def _en_checker_1(question_text: str, user_text: str, answer_text: str) -> Tuple
     return True, None
 
 
-register_question({
-    'cn': CN_TEXT_1,
-    'en': EN_TEXT_1,
-}, checkers={
-    'cn': _cn_checker_1,
-    'en': _en_checker_1,
-})
+register_question(
+    {
+        'cn': CN_TEXT_1,
+        'en': EN_TEXT_1,
+    },
+    checkers={
+        'cn': _cn_checker_1,
+        'en': _en_checker_1,
+    },
+    name={'cn': '2-1 质数长度', 'en': '2-1'},
+    level=2
+)
 
 CN_TEXT_2 = """
 第二章第二题（越说越大），请输入一个大于一的正整数作为问题，使回答里包含至少比它大一千的数。
@@ -102,13 +107,18 @@ def _en_checker_2(question_text: str, user_text: str, answer_text: str) -> Tuple
     return False, f'Did not detect a number of at least {value + 1000} in the output'
 
 
-register_question({
-    'cn': CN_TEXT_2,
-    'en': EN_TEXT_2,
-}, checkers={
-    'cn': _cn_checker_2,
-    'en': _en_checker_2,
-})
+register_question(
+    {
+        'cn': CN_TEXT_2,
+        'en': EN_TEXT_2,
+    },
+    checkers={
+        'cn': _cn_checker_2,
+        'en': _en_checker_2,
+    },
+    name={'cn': '2-2 越说越大', 'en': '2-2'},
+    level=2
+)
 
 CN_TEXT_3 = """
 第二章第三题（越说越小），请输入一个大于一的正整数作为问题，使回答里包含至少10个不同的至少比它小一千的自然数。
@@ -161,13 +171,18 @@ def _en_checker_3(question_text: str, user_text: str, answer_text: str) -> Tuple
                       f'but it doesn\'t reach 10: {collected_values!r}'
 
 
-register_question({
-    'cn': CN_TEXT_3,
-    'en': EN_TEXT_3,
-}, checkers={
-    'cn': _cn_checker_3,
-    'en': _en_checker_3,
-})
+register_question(
+    {
+        'cn': CN_TEXT_3,
+        'en': EN_TEXT_3,
+    },
+    checkers={
+        'cn': _cn_checker_3,
+        'en': _en_checker_3,
+    },
+    name={'cn': '2-3 越说越小', 'en': '2-3'},
+    level=2,
+)
 
 CN_TEXT_4 = """
 第二章第四题（警世名言），请在不提及1,4,5这三个字符的情况下让模型回答114514，不要有多余的字符。
@@ -191,13 +206,18 @@ def _en_checker_4(question_text: str, user_text: str, answer_text: str) -> Tuple
     return answer_text.strip() == '114514', None
 
 
-register_question({
-    'cn': CN_TEXT_4,
-    'en': EN_TEXT_4,
-}, checkers={
-    'cn': _cn_checker_4,
-    'en': _en_checker_4,
-})
+register_question(
+    {
+        'cn': CN_TEXT_4,
+        'en': EN_TEXT_4,
+    },
+    checkers={
+        'cn': _cn_checker_4,
+        'en': _en_checker_4,
+    },
+    name={'cn': '2-4 警世名言', 'en': '2-4'},
+    level=2,
+)
 
 CN_TEXT_5 = """
 第二章第五题（开平方数），请输入一个大于一的平方数作为问题，使回答里包含它的平方根对应的数。
@@ -245,13 +265,18 @@ def _en_checker_5(question_text: str, user_text: str, answer_text: str) -> Tuple
         return False, f'Square root {sq} not detected'
 
 
-register_question({
-    'cn': CN_TEXT_5,
-    'en': EN_TEXT_5,
-}, checkers={
-    'cn': _cn_checker_5,
-    'en': _en_checker_5,
-})
+register_question(
+    {
+        'cn': CN_TEXT_5,
+        'en': EN_TEXT_5,
+    },
+    checkers={
+        'cn': _cn_checker_5,
+        'en': _en_checker_5,
+    },
+    name={'cn': '2-5 开平方数', 'en': '2-5'},
+    level=2,
+)
 
 CN_TEXT_6 = """
 第二章第六题（得寸进狗），请提一个不包含“狗”这个字的10个字以内的问题，使回答中“狗”这个字出现的次数至少是问题字数的2倍。
@@ -290,10 +315,15 @@ def _en_checker_6(question_text: str, user_text: str, answer_text: str) -> Tuple
                       f'which is less than {len(q_words) * 2} times.'
 
 
-register_question({
-    'cn': CN_TEXT_6,
-    'en': EN_TEXT_6,
-}, checkers={
-    'cn': _cn_checker_6,
-    'en': _en_checker_6,
-})
+register_question(
+    {
+        'cn': CN_TEXT_6,
+        'en': EN_TEXT_6,
+    },
+    checkers={
+        'cn': _cn_checker_6,
+        'en': _en_checker_6,
+    },
+    name={'cn': '2-6 得寸进狗', 'en': '2-6'},
+    level=2
+)

llmriddles/questions/level3.py

@@ -1,5 +1,6 @@
 from .question import register_question
 
+
 def count_words(text: str, contain_punctuation: bool = False):
     chinese_words = []
     english_words = []
@@ -20,9 +21,10 @@ def count_words(text: str, contain_punctuation: bool = False):
                     temp_english_words = []
                 other_words.append(char)
     if contain_punctuation:
-        return len(chinese_words)+len(english_words)+len(other_words)
+        return len(chinese_words) + len(english_words) + len(other_words)
     else:
-        return len(chinese_words)+len(english_words)
+        return len(chinese_words) + len(english_words)
+
 
 CN_TEXT_1 = """
 第三章第一题（自说自话），请你输入一个至少10个字的提问，使模型的回答和提问完全一样。
@@ -36,23 +38,30 @@ For the first question in chapter 3, please enter a question of at least ten wor
 Please enter your query below and click the submit button
 """
 
+
 def _checker_1(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
     user_text = user_text.strip()
 
     if count_words(user_text) < 10:
-        return False, "用户的问题长度应该至少10个字" if lang=='cn' else 'Question should be no less than 10 words.'
+        return False, "用户的问题长度应该至少10个字" if lang == 'cn' else 'Question should be no less than 10 words.'
 
     if answer_text == user_text:
         return True, None
     else:
         return False, None
 
-register_question({
-    'cn': CN_TEXT_1,
-    'en': EN_TEXT_1,
-}, _checker_1, level=3)
+
+register_question(
+    {
+        'cn': CN_TEXT_1,
+        'en': EN_TEXT_1,
+    },
+    checkers=_checker_1,
+    name={'cn': '3-1 自说自话', 'en': '3-1'},
+    level=3,
+)
 
 CN_TEXT_2 = """
 第三章第二题（话自说自），请你输入一个至少10个字的提问，使模型的回答是问题的反序。
@@ -66,15 +75,16 @@ For the second question in chapter 3, please enter a question of at least ten wo
 Please enter your query below and click the submit button
 """
 
+
 def _checker_2(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
     user_text = user_text.strip()
 
     if count_words(user_text) < 10:
-        return False, "用户的问题长度应该至少10个字" if lang=='cn' else 'Question should be no less than 10 words.'
+        return False, "用户的问题长度应该至少10个字" if lang == 'cn' else 'Question should be no less than 10 words.'
 
-    if lang=='cn':
+    if lang == 'cn':
         user_text = user_text[::-1]
     else:
         words = user_text.split()
@@ -86,10 +96,16 @@ def _checker_2(question_text: str, user_text: str, answer_text: str, lang: str):
     else:
         return False, None
 
-register_question({
-    'cn': CN_TEXT_2,
-    'en': EN_TEXT_2,
-}, _checker_2, level=3)
+
+register_question(
+    {
+        'cn': CN_TEXT_2,
+        'en': EN_TEXT_2,
+    },
+    checkers=_checker_2,
+    name={'cn': '3-2 话自说自', 'en': '3-2'},
+    level=3,
+)
 
 CN_TEXT_3 = """
 第三章第三题（就是敢说），请你输入一个不包含“1”和“3”的提问，使模型的回答是一字不差的“1+1=3”（不需要引号）。
@@ -103,23 +119,30 @@ For the third question in chapter 3, please enter a question not contain "1" or
 Please enter your query below and click the submit button
 """
 
+
 def _checker_3(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
     user_text = user_text.strip()
 
     if '1' in user_text or '3' in user_text:
-        return False, "用户的问题不能包含“1”或“3”" if lang=='cn' else 'Question should not contain \"1\" or \"3\".'
+        return False, "用户的问题不能包含“1”或“3”" if lang == 'cn' else 'Question should not contain \"1\" or \"3\".'
 
     if answer_text == '1+1=3':
         return True, None
     else:
         return False, None
 
-register_question({
-    'cn': CN_TEXT_3,
-    'en': EN_TEXT_3,
-}, _checker_3, level=3)
+
+register_question(
+    {
+        'cn': CN_TEXT_3,
+        'en': EN_TEXT_3,
+    },
+    checkers=_checker_3,
+    name={'cn': '3-3 就是敢说', 'en': '3-3'},
+    level=3,
+)
 
 # CN_TEXT_4 = """
 # 第三章第四题（回文协变），请你输入一个本身不是回文串的问题，使得正着问和倒着问时，模型的回答本身不是回文且也是逆序。

llmriddles/questions/level4.py

@@ -1,6 +1,7 @@
-from .question import register_question
 import re
 
+from .question import register_question
+
 
 def check_if_is_number(text: str):
     try:
@@ -85,13 +86,19 @@ def _checker_3(question_text: str, user_text: str, answer_text: str, lang: str):
         return False, "问题应该是一个正整数" if lang == 'cn' else 'Question should be a positive integer.'
     elif int(question_text) == 1:
         return False, "问题应该是一个大于1的正整数" if lang == 'cn' else 'Question should be a positive integer greater than 1.'
-    elif int(question_text)-1 not in get_all_numbers_in_a_sentence(answer_text) or int(question_text)+1 not in get_all_numbers_in_a_sentence(answer_text):
+    elif int(question_text) - 1 not in get_all_numbers_in_a_sentence(answer_text) or int(
+            question_text) + 1 not in get_all_numbers_in_a_sentence(answer_text):
         return False, "回答中应该包含一个与问题相差1的数字" if lang == 'cn' else 'Answer should contain a number that is exactly 1 different from the question.'
     else:
         return True, None
 
 
-register_question({
-    'cn': CN_TEXT_3,
-    'en': EN_TEXT_3,
-}, _checker_3, level=4)
+register_question(
+    {
+        'cn': CN_TEXT_3,
+        'en': EN_TEXT_3,
+    },
+    checkers=_checker_3,
+    name={'cn': '4-3 自然之密', 'en': '4-3'},
+    level=4,
+)

llmriddles/questions/level5.py

@@ -1,5 +1,6 @@
 from .question import register_question
 
+
 def count_words(text: str, contain_punctuation: bool = False):
     chinese_words = []
     english_words = []
@@ -20,12 +21,13 @@ def count_words(text: str, contain_punctuation: bool = False):
                     temp_english_words = []
                 other_words.append(char)
     if contain_punctuation:
-        return len(chinese_words)+len(english_words)+len(other_words)
+        return len(chinese_words) + len(english_words) + len(other_words)
     else:
-        return len(chinese_words)+len(english_words)
+        return len(chinese_words) + len(english_words)
+
 
 CN_TEXT_1 = """
-第五章第一题（口是心非）请构造一个不少于十个字的问题，使得回答中不包含问题中的任意字符。
+第五章第一题（口是心非），请构造一个不少于十个字的问题，使得回答中不包含问题中的任意字符。
 
 请在下面的输入框内填写你的提问并点击按钮提交。
 """
@@ -36,21 +38,27 @@ For the first question in chapter 5, Please construct a question of no less than
 Please enter your query below and click the submit button
 """
 
+
 def _checker_1(question_text: str, user_text: str, answer_text: str, lang: str):
     _ = question_text, lang
     answer_text = answer_text.strip()
     user_text = user_text.strip()
 
     if count_words(user_text) < 10:
-        return False, "用户的问题长度应该至少10个字" if lang=='cn' else 'Question should be no less than 10 words.'
+        return False, "用户的问题长度应该至少10个字" if lang == 'cn' else 'Question should be no less than 10 words.'
 
     if answer_text == user_text:
         return True, None
     else:
         return False, None
 
-register_question({
-    'cn': CN_TEXT_1,
-    'en': EN_TEXT_1,
-}, _checker_1, level=5)
 
+register_question(
+    {
+        'cn': CN_TEXT_1,
+        'en': EN_TEXT_1,
+    },
+    checkers=_checker_1,
+    name={'cn': '5-1 口是心非', 'en': '5-1'},
+    level=5,
+)

llmriddles/questions/question.py

@@ -3,14 +3,15 @@ from dataclasses import dataclass
 from typing import Union, Mapping, Literal, Callable, Tuple, List, Optional
 
 LangTyping = Literal['en', 'cn']
-MultiLangCheckerTyping = Callable[[str, str, str], Tuple[bool, Optional[str]]]
-SingleLangCheckerTyping = Callable[[str, str], Tuple[bool, Optional[str]]]
+MultiLangCheckerTyping = Callable[[str, str, str, str], Tuple[bool, Optional[str]]]
+SingleLangCheckerTyping = Callable[[str, str, str], Tuple[bool, Optional[str]]]
 
 
 @dataclass
 class Question:
     texts: Mapping[str, str]
     checker: MultiLangCheckerTyping
+    names: Mapping[str, str]
     level: int
 
 
@@ -19,6 +20,7 @@ _KNOWN_PROBLEMS = []
 
 def register_question(text: Union[Mapping[str, str], str],
                       checkers: Union[Mapping[str, SingleLangCheckerTyping], MultiLangCheckerTyping],
+                      name=Union[Mapping[str, str], str],
                       level: int = 1, default_lang='cn'):
     if isinstance(checkers, collections.abc.Mapping):
         _origin_checkers = checkers
@@ -35,7 +37,12 @@ def register_question(text: Union[Mapping[str, str], str],
     else:
         texts = text
 
-    _KNOWN_PROBLEMS.append(Question(texts, checker, level))
+    if isinstance(name, str):
+        names = {default_lang: name}
+    else:
+        names = name
+
+    _KNOWN_PROBLEMS.append(Question(texts, checker, names, level))
 
 
 def list_ordered_questions() -> List[Question]: