Bloom-Medical-QA-LoRA

This model is a fine-tuned version of bigscience/bloom-560m on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.9682

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0168	10	2.5135
No log	0.0337	20	2.3298
No log	0.0505	30	2.2756
No log	0.0673	40	2.2419
No log	0.0842	50	2.2270
No log	0.1010	60	2.2102
No log	0.1178	70	2.1931
No log	0.1347	80	2.1846
No log	0.1515	90	2.1749
2.2471	0.1684	100	2.1645
2.2471	0.1852	110	2.1594
2.2471	0.2020	120	2.1557
2.2471	0.2189	130	2.1526
2.2471	0.2357	140	2.1495
2.2471	0.2525	150	2.1498
2.2471	0.2694	160	2.1401
2.2471	0.2862	170	2.1374
2.2471	0.3030	180	2.1317
2.2471	0.3199	190	2.1259
2.1174	0.3367	200	2.1248
2.1174	0.3535	210	2.1240
2.1174	0.3704	220	2.1179
2.1174	0.3872	230	2.1162
2.1174	0.4040	240	2.1163
2.1174	0.4209	250	2.1079
2.1174	0.4377	260	2.1075
2.1174	0.4545	270	2.1035
2.1174	0.4714	280	2.1030
2.1174	0.4882	290	2.0973
2.0744	0.5051	300	2.0895
2.0744	0.5219	310	2.0886
2.0744	0.5387	320	2.0894
2.0744	0.5556	330	2.0908
2.0744	0.5724	340	2.0841
2.0744	0.5892	350	2.0799
2.0744	0.6061	360	2.0799
2.0744	0.6229	370	2.0755
2.0744	0.6397	380	2.0753
2.0744	0.6566	390	2.0713
2.054	0.6734	400	2.0713
2.054	0.6902	410	2.0704
2.054	0.7071	420	2.0678
2.054	0.7239	430	2.0664
2.054	0.7407	440	2.0700
2.054	0.7576	450	2.0672
2.054	0.7744	460	2.0639
2.054	0.7912	470	2.0624
2.054	0.8081	480	2.0630
2.054	0.8249	490	2.0579
2.0228	0.8418	500	2.0548
2.0228	0.8586	510	2.0517
2.0228	0.8754	520	2.0490
2.0228	0.8923	530	2.0509
2.0228	0.9091	540	2.0488
2.0228	0.9259	550	2.0482
2.0228	0.9428	560	2.0446
2.0228	0.9596	570	2.0407
2.0228	0.9764	580	2.0445
2.0228	0.9933	590	2.0410
2.0186	1.0101	600	2.0426
2.0186	1.0269	610	2.0408
2.0186	1.0438	620	2.0407
2.0186	1.0606	630	2.0404
2.0186	1.0774	640	2.0363
2.0186	1.0943	650	2.0358
2.0186	1.1111	660	2.0365
2.0186	1.1279	670	2.0348
2.0186	1.1448	680	2.0343
2.0186	1.1616	690	2.0319
1.9341	1.1785	700	2.0268
1.9341	1.1953	710	2.0287
1.9341	1.2121	720	2.0302
1.9341	1.2290	730	2.0298
1.9341	1.2458	740	2.0307
1.9341	1.2626	750	2.0282
1.9341	1.2795	760	2.0305
1.9341	1.2963	770	2.0240
1.9341	1.3131	780	2.0245
1.9341	1.3300	790	2.0204
1.9309	1.3468	800	2.0208
1.9309	1.3636	810	2.0189
1.9309	1.3805	820	2.0179
1.9309	1.3973	830	2.0185
1.9309	1.4141	840	2.0187
1.9309	1.4310	850	2.0191
1.9309	1.4478	860	2.0218
1.9309	1.4646	870	2.0131
1.9309	1.4815	880	2.0096
1.9309	1.4983	890	2.0092
1.919	1.5152	900	2.0070
1.919	1.5320	910	2.0060
1.919	1.5488	920	2.0070
1.919	1.5657	930	2.0094
1.919	1.5825	940	2.0137
1.919	1.5993	950	2.0092
1.919	1.6162	960	2.0033
1.919	1.6330	970	2.0045
1.919	1.6498	980	2.0053
1.919	1.6667	990	2.0043
1.9546	1.6835	1000	2.0028
1.9546	1.7003	1010	2.0018
1.9546	1.7172	1020	2.0005
1.9546	1.7340	1030	1.9999
1.9546	1.7508	1040	1.9983
1.9546	1.7677	1050	1.9974
1.9546	1.7845	1060	1.9971
1.9546	1.8013	1070	1.9961
1.9546	1.8182	1080	1.9943
1.9546	1.8350	1090	1.9938
1.9247	1.8519	1100	1.9949
1.9247	1.8687	1110	1.9949
1.9247	1.8855	1120	1.9926
1.9247	1.9024	1130	1.9888
1.9247	1.9192	1140	1.9894
1.9247	1.9360	1150	1.9893
1.9247	1.9529	1160	1.9912
1.9247	1.9697	1170	1.9896
1.9247	1.9865	1180	1.9896
1.9247	2.0034	1190	1.9900
1.8978	2.0202	1200	1.9886
1.8978	2.0370	1210	1.9898
1.8978	2.0539	1220	1.9883
1.8978	2.0707	1230	1.9869
1.8978	2.0875	1240	1.9876
1.8978	2.1044	1250	1.9873
1.8978	2.1212	1260	1.9893
1.8978	2.1380	1270	1.9879
1.8978	2.1549	1280	1.9880
1.8978	2.1717	1290	1.9882
1.8637	2.1886	1300	1.9869
1.8637	2.2054	1310	1.9879
1.8637	2.2222	1320	1.9881
1.8637	2.2391	1330	1.9901
1.8637	2.2559	1340	1.9874
1.8637	2.2727	1350	1.9855
1.8637	2.2896	1360	1.9871
1.8637	2.3064	1370	1.9871
1.8637	2.3232	1380	1.9849
1.8637	2.3401	1390	1.9837
1.8206	2.3569	1400	1.9841
1.8206	2.3737	1410	1.9828
1.8206	2.3906	1420	1.9809
1.8206	2.4074	1430	1.9774
1.8206	2.4242	1440	1.9770
1.8206	2.4411	1450	1.9779
1.8206	2.4579	1460	1.9779
1.8206	2.4747	1470	1.9783
1.8206	2.4916	1480	1.9763
1.8206	2.5084	1490	1.9766
1.8587	2.5253	1500	1.9762
1.8587	2.5421	1510	1.9768
1.8587	2.5589	1520	1.9782
1.8587	2.5758	1530	1.9787
1.8587	2.5926	1540	1.9770
1.8587	2.6094	1550	1.9755
1.8587	2.6263	1560	1.9753
1.8587	2.6431	1570	1.9753
1.8587	2.6599	1580	1.9747
1.8587	2.6768	1590	1.9742
1.8226	2.6936	1600	1.9743
1.8226	2.7104	1610	1.9729
1.8226	2.7273	1620	1.9735
1.8226	2.7441	1630	1.9749
1.8226	2.7609	1640	1.9737
1.8226	2.7778	1650	1.9731
1.8226	2.7946	1660	1.9723
1.8226	2.8114	1670	1.9721
1.8226	2.8283	1680	1.9713
1.8226	2.8451	1690	1.9698
1.8331	2.8620	1700	1.9695
1.8331	2.8788	1710	1.9688
1.8331	2.8956	1720	1.9689
1.8331	2.9125	1730	1.9686
1.8331	2.9293	1740	1.9685
1.8331	2.9461	1750	1.9681
1.8331	2.9630	1760	1.9675
1.8331	2.9798	1770	1.9669
1.8331	2.9966	1780	1.9671
1.8331	3.0135	1790	1.9680
1.8251	3.0303	1800	1.9696
1.8251	3.0471	1810	1.9705
1.8251	3.0640	1820	1.9695
1.8251	3.0808	1830	1.9700
1.8251	3.0976	1840	1.9701
1.8251	3.1145	1850	1.9701
1.8251	3.1313	1860	1.9705
1.8251	3.1481	1870	1.9707
1.8251	3.1650	1880	1.9709
1.8251	3.1818	1890	1.9712
1.7809	3.1987	1900	1.9713
1.7809	3.2155	1910	1.9718
1.7809	3.2323	1920	1.9720
1.7809	3.2492	1930	1.9714
1.7809	3.2660	1940	1.9704
1.7809	3.2828	1950	1.9694
1.7809	3.2997	1960	1.9694
1.7809	3.3165	1970	1.9692
1.7809	3.3333	1980	1.9690
1.7809	3.3502	1990	1.9689
1.7957	3.3670	2000	1.9687
1.7957	3.3838	2010	1.9685
1.7957	3.4007	2020	1.9682
1.7957	3.4175	2030	1.9684
1.7957	3.4343	2040	1.9688
1.7957	3.4512	2050	1.9689
1.7957	3.4680	2060	1.9685
1.7957	3.4848	2070	1.9683
1.7957	3.5017	2080	1.9685
1.7957	3.5185	2090	1.9686
1.7859	3.5354	2100	1.9688
1.7859	3.5522	2110	1.9691
1.7859	3.5690	2120	1.9692
1.7859	3.5859	2130	1.9693
1.7859	3.6027	2140	1.9694
1.7859	3.6195	2150	1.9692
1.7859	3.6364	2160	1.9690
1.7859	3.6532	2170	1.9689
1.7859	3.6700	2180	1.9689
1.7859	3.6869	2190	1.9689
1.7794	3.7037	2200	1.9687
1.7794	3.7205	2210	1.9686
1.7794	3.7374	2220	1.9685
1.7794	3.7542	2230	1.9685
1.7794	3.7710	2240	1.9684
1.7794	3.7879	2250	1.9684
1.7794	3.8047	2260	1.9683
1.7794	3.8215	2270	1.9683
1.7794	3.8384	2280	1.9682
1.7794	3.8552	2290	1.9682
1.7766	3.8721	2300	1.9682
1.7766	3.8889	2310	1.9682
1.7766	3.9057	2320	1.9682
1.7766	3.9226	2330	1.9682
1.7766	3.9394	2340	1.9682
1.7766	3.9562	2350	1.9682
1.7766	3.9731	2360	1.9682
1.7766	3.9899	2370	1.9682

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 1.13.1+cu117
• Datasets 2.19.2
• Tokenizers 0.19.1