Codellama-7b-lora-rps-adapter

This model is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.3110

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
0.2019	2.5990	17000	0.2968
0.1928	2.6143	17100	0.2975
0.1992	2.6296	17200	0.2981
0.1975	2.6449	17300	0.2987
0.2003	2.6601	17400	0.2963
0.1847	2.6754	17500	0.2970
0.1945	2.6907	17600	0.2961
0.2057	2.7060	17700	0.2970
0.1782	2.7213	17800	0.2967
0.1813	2.7366	17900	0.2975
0.2001	2.7519	18000	0.2953
0.2074	2.7672	18100	0.2959
0.1957	2.7824	18200	0.2969
0.2006	2.7977	18300	0.2943
0.2021	2.8130	18400	0.2939
0.1862	2.8283	18500	0.2931
0.1951	2.8436	18600	0.2934
0.205	2.8589	18700	0.2936
0.2094	2.8742	18800	0.2919
0.1766	2.8895	18900	0.2935
0.2001	2.9048	19000	0.2931
0.1977	2.9200	19100	0.2941
0.1884	2.9353	19200	0.2922
0.1784	2.9506	19300	0.2927
0.1857	2.9659	19400	0.2921
0.1972	2.9812	19500	0.2926
0.1921	2.9965	19600	0.2929
0.1433	3.0118	19700	0.3114
0.1486	3.0271	19800	0.3115
0.1381	3.0423	19900	0.3147
0.1375	3.0576	20000	0.3122
0.1359	3.0729	20100	0.3144
0.133	3.0882	20200	0.3165
0.1346	3.1035	20300	0.3151
0.132	3.1188	20400	0.3169
0.1338	3.1341	20500	0.3137
0.1238	3.1494	20600	0.3160
0.1264	3.1647	20700	0.3146
0.1382	3.1799	20800	0.3139
0.136	3.1952	20900	0.3110
0.1321	3.2105	21000	0.3129
0.134	3.2258	21100	0.3148
0.134	3.2411	21200	0.3139
0.1338	3.2564	21300	0.3140
0.1317	3.2717	21400	0.3148
0.1281	3.2870	21500	0.3132
0.1279	3.3022	21600	0.3124
0.1355	3.3175	21700	0.3133
0.127	3.3328	21800	0.3129
0.1388	3.3481	21900	0.3157
0.1316	3.3634	22000	0.3134
0.1378	3.3787	22100	0.3127
0.1357	3.3940	22200	0.3131
0.1271	3.4093	22300	0.3141
0.1333	3.4246	22400	0.3142
0.1311	3.4398	22500	0.3133
0.1261	3.4551	22600	0.3138
0.1313	3.4704	22700	0.3129
0.1296	3.4857	22800	0.3135
0.1348	3.5010	22900	0.3134
0.1252	3.5163	23000	0.3131
0.1403	3.5316	23100	0.3117
0.1266	3.5469	23200	0.3126
0.135	3.5621	23300	0.3135
0.1344	3.5774	23400	0.3133
0.1452	3.5927	23500	0.3128
0.1285	3.6080	23600	0.3131
0.1235	3.6233	23700	0.3108
0.1255	3.6386	23800	0.3111
0.1335	3.6539	23900	0.3114
0.1397	3.6692	24000	0.3109
0.1359	3.6845	24100	0.3108
0.1269	3.6997	24200	0.3120
0.1345	3.7150	24300	0.3115
0.131	3.7303	24400	0.3111
0.1332	3.7456	24500	0.3115
0.1226	3.7609	24600	0.3123
0.1244	3.7762	24700	0.3114
0.123	3.7915	24800	0.3115
0.1302	3.8068	24900	0.3103
0.1291	3.8220	25000	0.3108
0.1335	3.8373	25100	0.3118
0.1251	3.8526	25200	0.3115
0.1321	3.8679	25300	0.3111
0.1249	3.8832	25400	0.3111
0.1324	3.8985	25500	0.3111
0.1236	3.9138	25600	0.3112
0.1399	3.9291	25700	0.3108
0.1255	3.9444	25800	0.3107
0.1462	3.9596	25900	0.3107
0.1217	3.9749	26000	0.3108
0.1238	3.9902	26100	0.3110

Framework versions

• PEFT 0.13.0
• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0