New instructions

README.md

@@ -24,63 +24,115 @@ ea0405cdb5bc638fee12de614f729ebc  consolidated.03.pth
 4babdbd05b8923226a9e9622492054b6  params.json
 ```
 
-These can be converted to HuggingFace Transformers-compatible weights using the script available [here](https://github.com/huggingface/transformers/blob/28f26c107b4a1c5c7e32ed4d9575622da0627a40/src/transformers/models/llama/convert_llama_weights_to_hf.py).
+**Important: Follow these exact steps to convert your original LLaMa checkpoint to a HuggingFace Transformers-compatible format. If you use the wrong versions of any dependency, you risk ending up with weights which are not compatible with the XOR files.**
 
-**Important**: It was tested with git version transformers 4.28.0.dev0 (git hash: **28f26c107b4a1c5c7e32ed4d9575622da0627a40**). Make sure the package tokenizers 0.13.3 is installed. Use of different versions may result in broken outputs.
+1. Create a clean Python **3.10** virtual environment & activate it:
 
 ```
-PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python convert_llama_weights_to_hf.py --input_dir ~/llama/  --output_dir ~/llama30b_hf/ --model_size 30B
+python3.10 -m venv xor_venv
+source xor_venv/bin/activate
 ```
 
-Run `find -type f -exec md5sum "{}" + > checklist.chk` in the conversion target directory. This should produce a `checklist.chk` with exactly the following content if your files are correct:
+2. Clone transformers repo and switch to tested version:
 
 ```
-d0e13331c103453e9e087d59dcf05432  ./pytorch_model-00001-of-00007.bin
-29aae4d31a0a4fe6906353001341d493  ./pytorch_model-00002-of-00007.bin
-b40838eb4e68e087b15b3d653ca1f5d7  ./pytorch_model-00003-of-00007.bin
-f845ecc481cb92b8a0586c2ce288b828  ./pytorch_model-00004-of-00007.bin
-f3b13d089840e6caf22cd6dd05b77ef0  ./pytorch_model-00005-of-00007.bin
-12e0d2d7a9c00c4237b1b0143c48a05e  ./pytorch_model-00007-of-00007.bin
-1348f7c8bb3ee4408b69305a10bdfafb  ./pytorch_model-00006-of-00007.bin
+git clone https://github.com/huggingface/transformers.git
+cd transformers
+git checkout d04ec99bec8a0b432fc03ed60cea9a1a20ebaf3c
+pip install .
+```
+
+3. Install **exactly** these dependency versions:
+
+```
+pip install torch==1.13.1 accelerate==0.18.0 sentencepiece==0.1.98 protobuf==3.20.1
+```
+
+4. Check `pip freeze` output:
+
+```
+accelerate==0.18.0
+certifi==2022.12.7
+charset-normalizer==3.1.0
+filelock==3.12.0
+huggingface-hub==0.13.4
+idna==3.4
+numpy==1.24.2
+nvidia-cublas-cu11==11.10.3.66
+nvidia-cuda-nvrtc-cu11==11.7.99
+nvidia-cuda-runtime-cu11==11.7.99
+nvidia-cudnn-cu11==8.5.0.96
+packaging==23.1
+protobuf==3.20.1
+psutil==5.9.5
+PyYAML==6.0
+regex==2023.3.23
+requests==2.28.2
+sentencepiece==0.1.98
+tokenizers==0.13.3
+torch==1.13.1
+tqdm==4.65.0
+transformers @ file:///mnt/data/koepf/transformers
+typing_extensions==4.5.0
+urllib3==1.26.15
+```
+
+5. While in `transformers` repo root, run HF LLaMA conversion script:
+
+```
+python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir <input_path_llama_base>  --output_dir <output_path_llama30b_hf> --model_size 30B
+```
+
+6. Run `find . -type f -exec md5sum "{}" +` in the conversion target directory (`output_dir`). This should produce exactly the following checksums if your files are correct:
+
+```
+462a2d07f65776f27c0facfa2affb9f9  ./pytorch_model-00007-of-00007.bin
+e1dc8c48a65279fb1fbccff14562e6a3  ./pytorch_model-00003-of-00007.bin
+9cffb1aeba11b16da84b56abb773d099  ./pytorch_model-00001-of-00007.bin
 aee09e21813368c49baaece120125ae3  ./generation_config.json
+92754d6c6f291819ffc3dfcaf470f541  ./pytorch_model-00005-of-00007.bin
+3eddc6fc02c0172d38727e5826181adb  ./pytorch_model-00004-of-00007.bin
 eeec4125e9c7560836b4873b6f8e3025  ./tokenizer.model
+99762d59efa6b96599e863893cf2da02  ./pytorch_model-00006-of-00007.bin
 598538f18fed1877b41f77de034c0c8a  ./config.json
 fdb311c39b8659a5d5c1991339bafc09  ./tokenizer.json
-b77e99aa2ddc3df500c2b2dc4455a6af  ./pytorch_model.bin.index.json
+fecfda4fba7bfd911e187a85db5fa2ef  ./pytorch_model.bin.index.json
 edd1a5897748864768b1fab645b31491  ./tokenizer_config.json
 6b2e0a735969660e720c27061ef3f3d3  ./special_tokens_map.json
+5cfcb78b908ffa02e681cce69dbe4303  ./pytorch_model-00002-of-00007.bin
 ```
 
-Once you have LLaMa weights in the correct format, you can apply the XOR decoding:
+**Important: You should now have the correct LLaMa weights and be ready to apply the XORs. If the checksums above do not match yours, there is a problem.**
+
+7. Once you have LLaMa weights in the correct format, you can apply the XOR decoding:
 
 ```
 python xor_codec.py oasst-sft-6-llama-30b/ oasst-sft-6-llama-30b-xor/ llama30b_hf/
 ```
 
-You should expect to see one warning message during execution:
+You should **expect to see one warning message** during execution:
 
 `Exception when processing 'added_tokens.json'`
 
-This is normal. If similar messages appear for other files, something has gone wrong.
+This is normal. **If similar messages appear for other files, something has gone wrong**.
 
-Now run `find -type f -exec md5sum "{}" + > checklist.chk` in the output directory (here `oasst-sft-6-llama-30b`). You should get a file with exactly these contents:
+8. Now run `find . -type f -exec md5sum "{}" +` in the output directory (here `oasst-sft-6-llama-30b`). You should get a file with exactly these checksums:
 
 ```
 970e99665d66ba3fad6fdf9b4910acc5  ./pytorch_model-00007-of-00007.bin
 659fcb7598dcd22e7d008189ecb2bb42  ./pytorch_model-00003-of-00007.bin
 ff6e4cf43ddf02fb5d3960f850af1220  ./pytorch_model-00001-of-00007.bin
 27b0dc092f99aa2efaf467b2d8026c3f  ./added_tokens.json
-aee09e21813368c49baaece120125ae3  ./generation_config.json
+2917a1cafb895cf57e746cfd7696bfe5  ./generation_config.json
 740c324ae65b1ec25976643cda79e479  ./pytorch_model-00005-of-00007.bin
 f7aefb4c63be2ac512fd905b45295235  ./pytorch_model-00004-of-00007.bin
 eeec4125e9c7560836b4873b6f8e3025  ./tokenizer.model
 369df2f0e38bda0d9629a12a77c10dfc  ./pytorch_model-00006-of-00007.bin
-27b9c7c8c62db80e92de14724f4950f3  ./config.json
-deb33dd4ffc3d2baddcce275a00b7c1b  ./tokenizer.json
+cc9dbf56b68b68a585cc7367696e06a7  ./config.json
 76d47e4f51a8df1d703c6f594981fcab  ./pytorch_model.bin.index.json
-ed59bfee4e87b9193fea5897d610ab24  ./tokenizer_config.json
-130f5e690becc2223f59384887c2a505  ./special_tokens_map.json
+fd9452959d711be29ccf04a97598e8d1  ./tokenizer_config.json
+785905630a0fe583122a8446a5abe287  ./special_tokens_map.json
 ae48c4c68e4e171d502dd0896aa19a84  ./pytorch_model-00002-of-00007.bin
 ```
 
-If so you have successfully decoded the weights and should be able to use the model with HuggingFace Transformers.
+If so you have successfully decoded the weights and should be able to use the model with HuggingFace Transformers. **If your checksums do not match those above, there is a problem.**