Update README.md

README.md

@@ -24,13 +24,16 @@ It achieves the following results on the evaluation set:
 
 See [config](https://huggingface.co/pszemraj/mega-small-2048-C1024-tk_id-simplewiki-MR50/blob/main/config.json) for architecture details. While not a ready 'pretrained' model, this was trained from scratch.
 
+This model uses the tokenizer from `roberta-base`.
+
 ## Intended uses & limitations
 
 More information needed
 
 ## Training and evaluation data
 
-- this was trained in `bf16`. the [official recommendation](https://github.com/facebookresearch/mega#tips) is fp32 - still exploring this.
+> **Note:** this was trained in `bf16`. the [official recommendation](https://github.com/facebookresearch/mega#tips) is fp32 - still exploring this.
+
 ## Training procedure
 
 ### Training hyperparameters
@@ -47,6 +50,12 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_ratio: 0.05
 - num_epochs: 3.0
 
+Additionally:
+
+- mask rate of 50%
+- whole-word masking
+
+
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss | Accuracy |