brucethemoose
commited on
Commit
•
be94b59
1
Parent(s):
1c175a6
Update README.md
Browse files
README.md
CHANGED
@@ -8,7 +8,7 @@ library_name: transformers
|
|
8 |
pipeline_tag: text-generation
|
9 |
---
|
10 |
|
11 |
-
**NousResearch/Nous-Capybara-34B**, **migtissera/Tess-M-v1.3** and **bhenrym14/airoboros-3_1-yi-34b-200k** merged with a new, experimental implementation of "dare ties" via mergekit.
|
12 |
|
13 |
Qantized with exllamav2 on 200 rows (400K tokens) on a long Orca-Vicuna format chat, a sci fi story and a fantasy story. This should hopefully yield better chat performance than the default wikitext quantization.
|
14 |
|
@@ -50,7 +50,7 @@ Second exllama quantization pass:
|
|
50 |
python convert.py --in_dir /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/capytessmes.json --cal_dataset /home/alpha/Documents/medium.parquet -l 2048 -r 200 -ml 2048 -mr 40 -gr 200 -ss 4096 -b 3.1 -hb 6 -cf /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties-exl2-4bpw-fiction -nr
|
51 |
```
|
52 |
|
53 |
-
dare_ties is testing with better perplexity than a regular ties merge with the same merge configuration. Model weights that add up to one also seem optimal from testing. And results seem... better than the previous dare merge with Tess 1.2?
|
54 |
|
55 |
I chose not to include other finetunes, such as Dolphin, because they aren't trained on the 200K base. If any other 200K finetunes pop up, let me know.
|
56 |
|
|
|
8 |
pipeline_tag: text-generation
|
9 |
---
|
10 |
|
11 |
+
**NousResearch/Nous-Capybara-34B**, **migtissera/Tess-M-v1.3** and **bhenrym14/airoboros-3_1-yi-34b-200k** merged with a new, experimental implementation of "dare ties" via mergekit.
|
12 |
|
13 |
Qantized with exllamav2 on 200 rows (400K tokens) on a long Orca-Vicuna format chat, a sci fi story and a fantasy story. This should hopefully yield better chat performance than the default wikitext quantization.
|
14 |
|
|
|
50 |
python convert.py --in_dir /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/capytessmes.json --cal_dataset /home/alpha/Documents/medium.parquet -l 2048 -r 200 -ml 2048 -mr 40 -gr 200 -ss 4096 -b 3.1 -hb 6 -cf /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties-exl2-4bpw-fiction -nr
|
51 |
```
|
52 |
|
53 |
+
dare_ties is testing with better perplexity than a regular ties merge with the same merge configuration. Model weights that add up to one also seem optimal from testing. And results at long context seem... better than the previous dare merge with Tess 1.2?
|
54 |
|
55 |
I chose not to include other finetunes, such as Dolphin, because they aren't trained on the 200K base. If any other 200K finetunes pop up, let me know.
|
56 |
|