Saving weights and logs at step 280000

README.md

@@ -17,22 +17,22 @@ datasets:
 Dataset:
 
 * [mC4 NL Cleaned](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned)
-* dataset split: full (33B tokens)
+* dataset config: full (33B tokens)
 
 Tokenizer:
 
-* New tokenizer trained on mC4 with the scripts from the Huggingface
+* Tokenizer trained on mC4 with scripts from the Huggingface
   Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
 
 Training details:
 
-* Trained for 240k steps (29 dec 2021)
+* Trained for 280k steps (30 dec 2021)
 * Block size: 512
 * Optimizer: adam, lr 8e-4, beta1 0.9, beta2 0.98
 * Warmup steps: 5000
 * Weight decay: 0.01
 
-Work in progress. Dec 2021.
+Work in progress. Dec 2021-Jan2022
 
 * Many thanks to the [Google TPU Research Cloud](https://sites.research.google/trc/about/) for providing access to a TPU cluster!
 * Thanks to @gsarti for creating the [t5-flax-gcp

flax_model.msgpack

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:65d0a6df749f03c6825305fe0a4ddf10af7019dfcc8da9a4e2777521606137f5
+oid sha256:d2bc942466bedf81fea88c9bbeaaafa7dfb2fec485a78c89c52705b841a2bf0a
 size 1419302302

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a9e66e988d4ce5517d9a3b59e2eb641274b0fce5815e8118f4a84f71259a09dc
+oid sha256:f6576b3366a236813f2b96767c8eb783a6d52ed8aa71222557e56edeae404cf0
 size 1444576537

runs/events.out.tfevents.1640332964.t1v-n-f9cfcc28-w-0.384322.0.v2

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a3905eb5358f312aee4fa9af05441f5c7554919cb9e3ff992c6f549447cca17a
-size 35774541
+oid sha256:c60bb6a82a55ceae1859f8fc81e83b0c19ee72e64de5ecdc95e012746328f4c6
+size 43681985