End of training

README.md

@@ -1,230 +1,57 @@
 ---
 base_model: codellama/CodeLlama-7b-Instruct-hf
-library_name: peft
-license: llama2
+library_name: transformers
+model_name: Codellama-7b-lora-rps-adapter
 tags:
+- generated_from_trainer
 - trl
 - sft
-- generated_from_trainer
-model-index:
-- name: Codellama-7b-lora-rps-adapter
-  results: []
+licence: license
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# Codellama-7b-lora-rps-adapter
-
-This model is a fine-tuned version of [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf) on the None dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.3066
+# Model Card for Codellama-7b-lora-rps-adapter
 
-## Model description
+This model is a fine-tuned version of [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf).
+It has been trained using [TRL](https://github.com/huggingface/trl).
 
-More information needed
+## Quick start
 
-## Intended uses & limitations
+```python
+from transformers import pipeline
 
-More information needed
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="SimonMA/Codellama-7b-lora-rps-adapter", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
 
-## Training and evaluation data
+## Training procedure
 
-More information needed
 
-## Training procedure
 
-### Training hyperparameters
+This model was trained with SFT.
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 2
-- eval_batch_size: 2
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 4
+### Framework versions
 
-### Training results
+- TRL: 0.12.0
+- Transformers: 4.46.2
+- Pytorch: 2.4.1+cu121
+- Datasets: 3.1.0
+- Tokenizers: 0.20.3
 
-| Training Loss | Epoch  | Step  | Validation Loss |
-|:-------------:|:------:|:-----:|:---------------:|
-| 0.197         | 2.6742 | 17000 | 0.2906          |
-| 0.1777        | 2.6821 | 17050 | 0.2934          |
-| 0.1949        | 2.6899 | 17100 | 0.2911          |
-| 0.2131        | 2.6978 | 17150 | 0.2928          |
-| 0.1839        | 2.7057 | 17200 | 0.2921          |
-| 0.2039        | 2.7135 | 17250 | 0.2896          |
-| 0.2187        | 2.7214 | 17300 | 0.2906          |
-| 0.185         | 2.7293 | 17350 | 0.2906          |
-| 0.1837        | 2.7371 | 17400 | 0.2933          |
-| 0.2117        | 2.7450 | 17450 | 0.2889          |
-| 0.2143        | 2.7529 | 17500 | 0.2904          |
-| 0.1814        | 2.7607 | 17550 | 0.2897          |
-| 0.1982        | 2.7686 | 17600 | 0.2898          |
-| 0.2243        | 2.7765 | 17650 | 0.2903          |
-| 0.1817        | 2.7843 | 17700 | 0.2895          |
-| 0.1921        | 2.7922 | 17750 | 0.2919          |
-| 0.2097        | 2.8001 | 17800 | 0.2913          |
-| 0.1883        | 2.8079 | 17850 | 0.2903          |
-| 0.1905        | 2.8158 | 17900 | 0.2882          |
-| 0.2034        | 2.8237 | 17950 | 0.2884          |
-| 0.2008        | 2.8315 | 18000 | 0.2891          |
-| 0.184         | 2.8394 | 18050 | 0.2883          |
-| 0.1732        | 2.8473 | 18100 | 0.2896          |
-| 0.1905        | 2.8551 | 18150 | 0.2895          |
-| 0.1812        | 2.8630 | 18200 | 0.2895          |
-| 0.1941        | 2.8709 | 18250 | 0.2899          |
-| 0.2063        | 2.8787 | 18300 | 0.2879          |
-| 0.1982        | 2.8866 | 18350 | 0.2868          |
-| 0.1946        | 2.8944 | 18400 | 0.2895          |
-| 0.2104        | 2.9023 | 18450 | 0.2874          |
-| 0.1851        | 2.9102 | 18500 | 0.2878          |
-| 0.1968        | 2.9180 | 18550 | 0.2868          |
-| 0.1964        | 2.9259 | 18600 | 0.2880          |
-| 0.1863        | 2.9338 | 18650 | 0.2880          |
-| 0.1875        | 2.9416 | 18700 | 0.2876          |
-| 0.1698        | 2.9495 | 18750 | 0.2863          |
-| 0.2082        | 2.9574 | 18800 | 0.2881          |
-| 0.1962        | 2.9652 | 18850 | 0.2869          |
-| 0.2061        | 2.9731 | 18900 | 0.2860          |
-| 0.2132        | 2.9810 | 18950 | 0.2869          |
-| 0.1854        | 2.9888 | 19000 | 0.2875          |
-| 0.1906        | 2.9967 | 19050 | 0.2879          |
-| 0.144         | 3.0046 | 19100 | 0.3005          |
-| 0.1302        | 3.0124 | 19150 | 0.3097          |
-| 0.1324        | 3.0203 | 19200 | 0.3090          |
-| 0.1344        | 3.0282 | 19250 | 0.3094          |
-| 0.1392        | 3.0360 | 19300 | 0.3064          |
-| 0.1464        | 3.0439 | 19350 | 0.3066          |
-| 0.141         | 3.0518 | 19400 | 0.3070          |
-| 0.1275        | 3.0596 | 19450 | 0.3103          |
-| 0.1284        | 3.0675 | 19500 | 0.3074          |
-| 0.1397        | 3.0754 | 19550 | 0.3111          |
-| 0.1335        | 3.0832 | 19600 | 0.3105          |
-| 0.1302        | 3.0911 | 19650 | 0.3082          |
-| 0.1315        | 3.0989 | 19700 | 0.3094          |
-| 0.128         | 3.1068 | 19750 | 0.3110          |
-| 0.1272        | 3.1147 | 19800 | 0.3094          |
-| 0.1227        | 3.1225 | 19850 | 0.3074          |
-| 0.1375        | 3.1304 | 19900 | 0.3093          |
-| 0.1344        | 3.1383 | 19950 | 0.3092          |
-| 0.1301        | 3.1461 | 20000 | 0.3098          |
-| 0.1339        | 3.1540 | 20050 | 0.3083          |
-| 0.1398        | 3.1619 | 20100 | 0.3100          |
-| 0.132         | 3.1697 | 20150 | 0.3109          |
-| 0.1499        | 3.1776 | 20200 | 0.3070          |
-| 0.1438        | 3.1855 | 20250 | 0.3075          |
-| 0.1267        | 3.1933 | 20300 | 0.3106          |
-| 0.1282        | 3.2012 | 20350 | 0.3082          |
-| 0.1365        | 3.2091 | 20400 | 0.3075          |
-| 0.1239        | 3.2169 | 20450 | 0.3110          |
-| 0.1507        | 3.2248 | 20500 | 0.3087          |
-| 0.1364        | 3.2327 | 20550 | 0.3112          |
-| 0.1281        | 3.2405 | 20600 | 0.3092          |
-| 0.1271        | 3.2484 | 20650 | 0.3104          |
-| 0.1124        | 3.2563 | 20700 | 0.3097          |
-| 0.1382        | 3.2641 | 20750 | 0.3111          |
-| 0.1415        | 3.2720 | 20800 | 0.3101          |
-| 0.1246        | 3.2798 | 20850 | 0.3115          |
-| 0.1337        | 3.2877 | 20900 | 0.3095          |
-| 0.1378        | 3.2956 | 20950 | 0.3069          |
-| 0.1219        | 3.3034 | 21000 | 0.3081          |
-| 0.1303        | 3.3113 | 21050 | 0.3098          |
-| 0.1445        | 3.3192 | 21100 | 0.3081          |
-| 0.134         | 3.3270 | 21150 | 0.3090          |
-| 0.1389        | 3.3349 | 21200 | 0.3098          |
-| 0.1388        | 3.3428 | 21250 | 0.3087          |
-| 0.1317        | 3.3506 | 21300 | 0.3094          |
-| 0.1367        | 3.3585 | 21350 | 0.3080          |
-| 0.1267        | 3.3664 | 21400 | 0.3092          |
-| 0.1333        | 3.3742 | 21450 | 0.3102          |
-| 0.1266        | 3.3821 | 21500 | 0.3102          |
-| 0.1345        | 3.3900 | 21550 | 0.3075          |
-| 0.1279        | 3.3978 | 21600 | 0.3083          |
-| 0.1342        | 3.4057 | 21650 | 0.3078          |
-| 0.141         | 3.4136 | 21700 | 0.3102          |
-| 0.1241        | 3.4214 | 21750 | 0.3066          |
-| 0.14          | 3.4293 | 21800 | 0.3083          |
-| 0.1232        | 3.4372 | 21850 | 0.3070          |
-| 0.1296        | 3.4450 | 21900 | 0.3081          |
-| 0.1286        | 3.4529 | 21950 | 0.3065          |
-| 0.1313        | 3.4608 | 22000 | 0.3071          |
-| 0.1484        | 3.4686 | 22050 | 0.3058          |
-| 0.1395        | 3.4765 | 22100 | 0.3074          |
-| 0.1311        | 3.4843 | 22150 | 0.3064          |
-| 0.1116        | 3.4922 | 22200 | 0.3095          |
-| 0.1269        | 3.5001 | 22250 | 0.3102          |
-| 0.1308        | 3.5079 | 22300 | 0.3067          |
-| 0.127         | 3.5158 | 22350 | 0.3077          |
-| 0.1176        | 3.5237 | 22400 | 0.3086          |
-| 0.1234        | 3.5315 | 22450 | 0.3095          |
-| 0.1359        | 3.5394 | 22500 | 0.3075          |
-| 0.1337        | 3.5473 | 22550 | 0.3083          |
-| 0.1224        | 3.5551 | 22600 | 0.3088          |
-| 0.1286        | 3.5630 | 22650 | 0.3090          |
-| 0.1341        | 3.5709 | 22700 | 0.3076          |
-| 0.1419        | 3.5787 | 22750 | 0.3099          |
-| 0.1478        | 3.5866 | 22800 | 0.3072          |
-| 0.1215        | 3.5945 | 22850 | 0.3080          |
-| 0.1298        | 3.6023 | 22900 | 0.3073          |
-| 0.1368        | 3.6102 | 22950 | 0.3071          |
-| 0.1388        | 3.6181 | 23000 | 0.3070          |
-| 0.1239        | 3.6259 | 23050 | 0.3069          |
-| 0.1202        | 3.6338 | 23100 | 0.3066          |
-| 0.1329        | 3.6417 | 23150 | 0.3060          |
-| 0.1262        | 3.6495 | 23200 | 0.3070          |
-| 0.1221        | 3.6574 | 23250 | 0.3084          |
-| 0.1233        | 3.6653 | 23300 | 0.3068          |
-| 0.1222        | 3.6731 | 23350 | 0.3063          |
-| 0.133         | 3.6810 | 23400 | 0.3067          |
-| 0.1276        | 3.6888 | 23450 | 0.3054          |
-| 0.1214        | 3.6967 | 23500 | 0.3065          |
-| 0.1308        | 3.7046 | 23550 | 0.3072          |
-| 0.1278        | 3.7124 | 23600 | 0.3074          |
-| 0.1177        | 3.7203 | 23650 | 0.3070          |
-| 0.1302        | 3.7282 | 23700 | 0.3067          |
-| 0.1279        | 3.7360 | 23750 | 0.3068          |
-| 0.132         | 3.7439 | 23800 | 0.3078          |
-| 0.143         | 3.7518 | 23850 | 0.3070          |
-| 0.1365        | 3.7596 | 23900 | 0.3068          |
-| 0.1456        | 3.7675 | 23950 | 0.3073          |
-| 0.1312        | 3.7754 | 24000 | 0.3069          |
-| 0.1304        | 3.7832 | 24050 | 0.3073          |
-| 0.1409        | 3.7911 | 24100 | 0.3069          |
-| 0.1369        | 3.7990 | 24150 | 0.3067          |
-| 0.1291        | 3.8068 | 24200 | 0.3065          |
-| 0.1114        | 3.8147 | 24250 | 0.3073          |
-| 0.1294        | 3.8226 | 24300 | 0.3067          |
-| 0.1223        | 3.8304 | 24350 | 0.3071          |
-| 0.1166        | 3.8383 | 24400 | 0.3074          |
-| 0.1233        | 3.8462 | 24450 | 0.3076          |
-| 0.1348        | 3.8540 | 24500 | 0.3073          |
-| 0.1203        | 3.8619 | 24550 | 0.3069          |
-| 0.1326        | 3.8697 | 24600 | 0.3072          |
-| 0.123         | 3.8776 | 24650 | 0.3070          |
-| 0.13          | 3.8855 | 24700 | 0.3069          |
-| 0.1305        | 3.8933 | 24750 | 0.3073          |
-| 0.1366        | 3.9012 | 24800 | 0.3075          |
-| 0.1428        | 3.9091 | 24850 | 0.3078          |
-| 0.1258        | 3.9169 | 24900 | 0.3075          |
-| 0.135         | 3.9248 | 24950 | 0.3073          |
-| 0.1282        | 3.9327 | 25000 | 0.3071          |
-| 0.1323        | 3.9405 | 25050 | 0.3073          |
-| 0.1242        | 3.9484 | 25100 | 0.3072          |
-| 0.1453        | 3.9563 | 25150 | 0.3071          |
-| 0.1441        | 3.9641 | 25200 | 0.3067          |
-| 0.1273        | 3.9720 | 25250 | 0.3066          |
-| 0.115         | 3.9799 | 25300 | 0.3067          |
-| 0.1337        | 3.9877 | 25350 | 0.3067          |
-| 0.1259        | 3.9956 | 25400 | 0.3066          |
+## Citations
 
 
-### Framework versions
 
-- PEFT 0.13.0
-- Transformers 4.45.1
-- Pytorch 2.4.1+cu121
-- Datasets 3.0.1
-- Tokenizers 0.20.0
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

adapter_config.json

@@ -20,13 +20,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "down_proj",
+    "v_proj",
     "o_proj",
-    "k_proj",
-    "q_proj",
     "gate_proj",
-    "v_proj",
-    "up_proj"
+    "up_proj",
+    "down_proj",
+    "k_proj",
+    "q_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:99d96065daed251701f196a470db6fbbc39963213624893f959bb9cab48b1b3f
+oid sha256:e300e9ce578ee5c5d293eb7a2081aba28c6d1693030e10d5649cc892d2c0c288
 size 2332095256

runs/Nov08_11-01-31_f9264325da9e/events.out.tfevents.1731063704.f9264325da9e.13873.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2d9b9212894bb1c1763ef0255e4e0deac34013a2d1d9b563d8c102349c599b3
+size 170730

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:34d9e83567d3beb472a14841c9158888c402a755601b0a9eac9fb43a1971c742
+oid sha256:32ae18e8fac07e67fc5ea615a995e7dd1f92b8d9f9bb32e00065f4d5db168749
 size 5560