BlinkDL
/

rwkv-4-pile-14b

Text Generation

Model card Files Files and versions Community

BlinkDL commited on Jan 22, 2023

Commit

b50c689

•

1 Parent(s): f2acbdb

Update README.md

Files changed (1) hide show

README.md +7 -7

README.md CHANGED Viewed

@@ -26,10 +26,10 @@ ctx_len = 1024
 n_layer = 40
 n_embd = 5120
-Preview checkpoint: RWKV-4-Pile-14B-20221231-4514.pth : Trained on the Pile for 186B tokens.
-* Pile loss 1.8014
-* LAMBADA ppl 3.92, acc 69.96%
-* PIQA acc 77.42%
-* SC2016 acc 74.18%
-* Hellaswag acc_norm 68.47%
-* WinoGrande acc 62.35%

 n_layer = 40
 n_embd = 5120
+Preview checkpoint: RWKV-4-Pile-14B-20230121-6210.pth : Trained on the Pile for 256B tokens.
+* Pile loss 1.7746
+* LAMBADA ppl 3.83, acc 70.79%
+* PIQA acc 77.53%
+* SC2016 acc 74.83%
+* Hellaswag acc_norm 69.78%
+* WinoGrande acc 63.54%