Commit
•
0f24f91
1
Parent(s):
c3a35dc
Add exact param counts (#106)
Browse files- Add exact param counts (ae814bfe70ae96fd90e5241be1d8837fec5c439e)
README.md
CHANGED
|
@@ -265,7 +265,9 @@ Please see [the BLOOM training README](https://github.com/bigscience-workshop/bi
|
|
| 265 |
|
| 266 |
* ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions
|
| 267 |
|
| 268 |
-
* 176
|
|
|
|
|
|
|
| 269 |
|
| 270 |
* 70 layers, 112 attention heads
|
| 271 |
|
|
|
|
| 265 |
|
| 266 |
* ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions
|
| 267 |
|
| 268 |
+
* 176,247,271,424 parameters:
|
| 269 |
+
|
| 270 |
+
* 3,596,615,680 embedding parameters
|
| 271 |
|
| 272 |
* 70 layers, 112 attention heads
|
| 273 |
|