|
--- |
|
base_model: codellama/CodeLlama-7b-Instruct-hf |
|
library_name: peft |
|
license: llama2 |
|
tags: |
|
- trl |
|
- sft |
|
- generated_from_trainer |
|
model-index: |
|
- name: Codellama-7b-lora-rps-adapter |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# Codellama-7b-lora-rps-adapter |
|
|
|
This model is a fine-tuned version of [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf) on the None dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.3110 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.0002 |
|
- train_batch_size: 2 |
|
- eval_batch_size: 2 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_ratio: 0.03 |
|
- num_epochs: 4 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|:-------------:|:------:|:-----:|:---------------:| |
|
| 0.2019 | 2.5990 | 17000 | 0.2968 | |
|
| 0.1928 | 2.6143 | 17100 | 0.2975 | |
|
| 0.1992 | 2.6296 | 17200 | 0.2981 | |
|
| 0.1975 | 2.6449 | 17300 | 0.2987 | |
|
| 0.2003 | 2.6601 | 17400 | 0.2963 | |
|
| 0.1847 | 2.6754 | 17500 | 0.2970 | |
|
| 0.1945 | 2.6907 | 17600 | 0.2961 | |
|
| 0.2057 | 2.7060 | 17700 | 0.2970 | |
|
| 0.1782 | 2.7213 | 17800 | 0.2967 | |
|
| 0.1813 | 2.7366 | 17900 | 0.2975 | |
|
| 0.2001 | 2.7519 | 18000 | 0.2953 | |
|
| 0.2074 | 2.7672 | 18100 | 0.2959 | |
|
| 0.1957 | 2.7824 | 18200 | 0.2969 | |
|
| 0.2006 | 2.7977 | 18300 | 0.2943 | |
|
| 0.2021 | 2.8130 | 18400 | 0.2939 | |
|
| 0.1862 | 2.8283 | 18500 | 0.2931 | |
|
| 0.1951 | 2.8436 | 18600 | 0.2934 | |
|
| 0.205 | 2.8589 | 18700 | 0.2936 | |
|
| 0.2094 | 2.8742 | 18800 | 0.2919 | |
|
| 0.1766 | 2.8895 | 18900 | 0.2935 | |
|
| 0.2001 | 2.9048 | 19000 | 0.2931 | |
|
| 0.1977 | 2.9200 | 19100 | 0.2941 | |
|
| 0.1884 | 2.9353 | 19200 | 0.2922 | |
|
| 0.1784 | 2.9506 | 19300 | 0.2927 | |
|
| 0.1857 | 2.9659 | 19400 | 0.2921 | |
|
| 0.1972 | 2.9812 | 19500 | 0.2926 | |
|
| 0.1921 | 2.9965 | 19600 | 0.2929 | |
|
| 0.1433 | 3.0118 | 19700 | 0.3114 | |
|
| 0.1486 | 3.0271 | 19800 | 0.3115 | |
|
| 0.1381 | 3.0423 | 19900 | 0.3147 | |
|
| 0.1375 | 3.0576 | 20000 | 0.3122 | |
|
| 0.1359 | 3.0729 | 20100 | 0.3144 | |
|
| 0.133 | 3.0882 | 20200 | 0.3165 | |
|
| 0.1346 | 3.1035 | 20300 | 0.3151 | |
|
| 0.132 | 3.1188 | 20400 | 0.3169 | |
|
| 0.1338 | 3.1341 | 20500 | 0.3137 | |
|
| 0.1238 | 3.1494 | 20600 | 0.3160 | |
|
| 0.1264 | 3.1647 | 20700 | 0.3146 | |
|
| 0.1382 | 3.1799 | 20800 | 0.3139 | |
|
| 0.136 | 3.1952 | 20900 | 0.3110 | |
|
| 0.1321 | 3.2105 | 21000 | 0.3129 | |
|
| 0.134 | 3.2258 | 21100 | 0.3148 | |
|
| 0.134 | 3.2411 | 21200 | 0.3139 | |
|
| 0.1338 | 3.2564 | 21300 | 0.3140 | |
|
| 0.1317 | 3.2717 | 21400 | 0.3148 | |
|
| 0.1281 | 3.2870 | 21500 | 0.3132 | |
|
| 0.1279 | 3.3022 | 21600 | 0.3124 | |
|
| 0.1355 | 3.3175 | 21700 | 0.3133 | |
|
| 0.127 | 3.3328 | 21800 | 0.3129 | |
|
| 0.1388 | 3.3481 | 21900 | 0.3157 | |
|
| 0.1316 | 3.3634 | 22000 | 0.3134 | |
|
| 0.1378 | 3.3787 | 22100 | 0.3127 | |
|
| 0.1357 | 3.3940 | 22200 | 0.3131 | |
|
| 0.1271 | 3.4093 | 22300 | 0.3141 | |
|
| 0.1333 | 3.4246 | 22400 | 0.3142 | |
|
| 0.1311 | 3.4398 | 22500 | 0.3133 | |
|
| 0.1261 | 3.4551 | 22600 | 0.3138 | |
|
| 0.1313 | 3.4704 | 22700 | 0.3129 | |
|
| 0.1296 | 3.4857 | 22800 | 0.3135 | |
|
| 0.1348 | 3.5010 | 22900 | 0.3134 | |
|
| 0.1252 | 3.5163 | 23000 | 0.3131 | |
|
| 0.1403 | 3.5316 | 23100 | 0.3117 | |
|
| 0.1266 | 3.5469 | 23200 | 0.3126 | |
|
| 0.135 | 3.5621 | 23300 | 0.3135 | |
|
| 0.1344 | 3.5774 | 23400 | 0.3133 | |
|
| 0.1452 | 3.5927 | 23500 | 0.3128 | |
|
| 0.1285 | 3.6080 | 23600 | 0.3131 | |
|
| 0.1235 | 3.6233 | 23700 | 0.3108 | |
|
| 0.1255 | 3.6386 | 23800 | 0.3111 | |
|
| 0.1335 | 3.6539 | 23900 | 0.3114 | |
|
| 0.1397 | 3.6692 | 24000 | 0.3109 | |
|
| 0.1359 | 3.6845 | 24100 | 0.3108 | |
|
| 0.1269 | 3.6997 | 24200 | 0.3120 | |
|
| 0.1345 | 3.7150 | 24300 | 0.3115 | |
|
| 0.131 | 3.7303 | 24400 | 0.3111 | |
|
| 0.1332 | 3.7456 | 24500 | 0.3115 | |
|
| 0.1226 | 3.7609 | 24600 | 0.3123 | |
|
| 0.1244 | 3.7762 | 24700 | 0.3114 | |
|
| 0.123 | 3.7915 | 24800 | 0.3115 | |
|
| 0.1302 | 3.8068 | 24900 | 0.3103 | |
|
| 0.1291 | 3.8220 | 25000 | 0.3108 | |
|
| 0.1335 | 3.8373 | 25100 | 0.3118 | |
|
| 0.1251 | 3.8526 | 25200 | 0.3115 | |
|
| 0.1321 | 3.8679 | 25300 | 0.3111 | |
|
| 0.1249 | 3.8832 | 25400 | 0.3111 | |
|
| 0.1324 | 3.8985 | 25500 | 0.3111 | |
|
| 0.1236 | 3.9138 | 25600 | 0.3112 | |
|
| 0.1399 | 3.9291 | 25700 | 0.3108 | |
|
| 0.1255 | 3.9444 | 25800 | 0.3107 | |
|
| 0.1462 | 3.9596 | 25900 | 0.3107 | |
|
| 0.1217 | 3.9749 | 26000 | 0.3108 | |
|
| 0.1238 | 3.9902 | 26100 | 0.3110 | |
|
|
|
|
|
### Framework versions |
|
|
|
- PEFT 0.13.0 |
|
- Transformers 4.45.1 |
|
- Pytorch 2.4.1+cu121 |
|
- Datasets 3.0.1 |
|
- Tokenizers 0.20.0 |
