Update README.md

README.md

@@ -2,7 +2,7 @@
 
 ## Model Description
 
-PT-Neo-125M-Code-Clippy-Dedup is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) finetuned using causal language modeling on our version of the Code Clippy Data dataset that has duplicates, which was scraped from public Github repositories (more information in the provided link). This model is specialized to autocomplete methods in multiple programming languages.
+GPT-Neo-125M-Code-Clippy is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) finetuned using causal language modeling on our version of the Code Clippy Data dataset that has duplicates, which was scraped from public Github repositories (more information in the provided link). This model is specialized to autocomplete methods in multiple programming languages. As discussed in OpenAI's [Codex paper](https://arxiv.org/abs/2107.03374), we modified the GPT-Neo model and tokenizer to accommodate for additional whitespace characters. Specifically, we add the following tokens `["\t\t", "  ", "    ", "        "]` and since they are all related to indentation, we initialize the embedding layer of these tokens with the same weights as the `\t` token already present in the model in hopes the model will learn to associate these whitespace characters with indentation faster.
 
 ## Training data
 
@@ -10,11 +10,48 @@ PT-Neo-125M-Code-Clippy-Dedup is a [GPT-Neo-125M model](https://huggingface.co/E
 
 ## Training procedure
 
-The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_apps.py).
+The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_streaming_flax.py).
+
+To reproduce the training one can use this command with the above script:
+```bash
+./run_clm_streaming_flax_v2.py \
+    --output_dir $HOME/gpt-neo-125M-code-clippy \
+    --model_name_or_path="flax-community/gpt-neo-125M-code-clippy" \
+    --dataset_name $HOME/gpt-code-clippy/code_clippy.py \
+    --data_dir /home/shared/code_clippy_data \
+    --text_column_name="text" \
+    --do_train --do_eval \
+    --block_size="2048" \
+    --per_device_train_batch_size="8" \
+    --per_device_eval_batch_size="16" \
+    --preprocessing_num_workers="8" \
+    --learning_rate="1e-4" \
+    --max_steps 100000 \
+    --warmup_steps 2500 \
+    --decay_steps 25000 \
+    --adam_beta1="0.9" \
+    --adam_beta2="0.95" \
+    --weight_decay="0.1" \
+    --overwrite_output_dir \
+    --logging_steps="100" \
+    --eval_steps="500" \
+    --push_to_hub="False" \
+    --report_to="all" \
+    --dtype="bfloat16" \
+    --skip_memory_metrics="True" \
+    --save_steps="500" \
+    --save_total_limit 10 \
+    --gradient_accumulation_steps 16 \
+    --report_to="wandb" \
+    --run_name="125m_1e-4lr_1024bs" \
+    --max_eval_samples 2000 \
+    --save_optimizer true
+```
 
 ## Intended Use and Limitations
 
-The model is finetuned text file from github repositories (mostly programming languages but also markdown and other project related files).
+The model is finetuned on text files from github repositories (mostly programming languages but also markdown and other project related files).
+
 
 ### How to use
 
@@ -24,7 +61,7 @@ You can use this model directly with a pipeline for text generation. This exampl
 
 from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM
 
-model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy-dedup")
+model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy")
 
 tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-125M-code-clippy")