Update README.md
Browse files
README.md
CHANGED
@@ -82,6 +82,14 @@ print(output[0]['generated_text'])
|
|
82 |
|
83 |
Note: If you want to use flash attention, call _AutoModelForCausalLM.from_pretrained()_ with _attn_implementation="flash_attention_2"_
|
84 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
85 |
## Demo
|
86 |
|
87 |
Here's a colab notebook with a chat interface, you can use this to interact with the chat model.
|
|
|
82 |
|
83 |
Note: If you want to use flash attention, call _AutoModelForCausalLM.from_pretrained()_ with _attn_implementation="flash_attention_2"_
|
84 |
|
85 |
+
|
86 |
+
## Benchmarks
|
87 |
+
|
88 |
+
These benchmarks were run using EleutherAI's [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
|
89 |
+
|
90 |
+
- **IFEval (Instruction Following Evaluation)**: IFEval is a fairly interesting dataset that tests the capability of models to clearly follow explicit instructions, such as “include keyword x” or “use format y”. The models are tested on their ability to strictly follow formatting instructions rather than the actual contents generated, allowing strict and rigorous metrics to be used.
|
91 |
+
- Score: **45.83**
|
92 |
+
|
93 |
## Demo
|
94 |
|
95 |
Here's a colab notebook with a chat interface, you can use this to interact with the chat model.
|