isarth/roberta-news-chatgpt

chatgpt-mlm

This model is a fine-tuned version of on the tmp/cnn_clean_chatgpt_data/ dataset. It achieves the following results on the evaluation set:

• Loss: 1.4969
• Accuracy: 0.7015

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 6
• num_epochs: 75.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
7.2061	0.2	500	6.6958	0.1161
6.6051	0.4	1000	6.5527	0.1303
6.5016	0.6	1500	6.4720	0.1382
6.4189	0.8	2000	6.3796	0.1424
6.3648	1.0	2500	6.3224	0.1448
6.2787	1.2	3000	6.2787	0.1411
6.2583	1.4	3500	6.2467	0.1446
6.2211	1.6	4000	6.2162	0.1475
6.1897	1.8	4500	6.1933	0.1466
6.1625	2.0	5000	6.1704	0.1483
6.1412	2.2	5500	6.1527	0.1484
6.1062	2.4	6000	6.1296	0.1492
6.1003	2.6	6500	6.1275	0.1483
6.0944	2.8	7000	6.0983	0.1496
6.077	3.0	7500	6.0839	0.1509
6.0419	3.2	8000	6.0747	0.1504
6.0264	3.4	8500	6.0729	0.1506
6.0222	3.6	9000	6.0585	0.1504
6.0067	3.8	9500	6.0518	0.1500
6.0045	4.0	10000	6.0300	0.1504
5.9659	4.2	10500	6.0248	0.1504
5.9542	4.4	11000	6.0143	0.1512
5.9479	4.6	11500	5.9891	0.1514
5.9506	4.8	12000	5.9827	0.1517
5.9358	5.0	12500	5.9973	0.1509
5.9114	5.2	13000	5.9761	0.1505
5.9089	5.4	13500	5.9637	0.1516
5.9008	5.6	14000	5.9535	0.1515
5.9007	5.8	14500	5.9343	0.1530
5.8734	6.0	15000	5.9255	0.1532
5.8519	6.2	15500	5.9213	0.1527
5.8383	6.4	16000	5.9126	0.1513
5.8461	6.6	16500	5.9041	0.1525
5.8387	6.8	17000	5.8923	0.1517
5.831	7.0	17500	5.8782	0.1558
5.8003	7.2	18000	5.8660	0.1554
5.7832	7.4	18500	5.8508	0.1560
5.7902	7.6	19000	5.8495	0.1558
5.7707	7.8	19500	5.8376	0.1553
5.7638	8.0	20000	5.8289	0.1564
5.741	8.2	20500	5.8230	0.1574
5.7291	8.4	21000	5.8110	0.1574
5.7206	8.6	21500	5.8014	0.1575
5.6974	8.8	22000	5.7644	0.1605
5.6954	9.0	22500	5.7404	0.1638
5.6467	9.2	23000	5.7040	0.1668
5.6134	9.4	23500	5.6656	0.1738
5.5855	9.6	24000	5.6262	0.1787
5.5374	9.8	24500	5.5587	0.1883
5.4678	10.0	25000	5.4388	0.2009
5.3324	10.2	25500	5.2703	0.2203
5.1849	10.4	26000	5.0908	0.2434
5.0273	10.6	26500	4.9103	0.2657
4.8718	10.8	27000	4.7637	0.2844
4.7523	11.0	27500	4.6064	0.3023
4.5814	11.2	28000	4.4398	0.3220
4.4627	11.4	28500	4.3005	0.3376
4.3228	11.6	29000	4.1771	0.3520
4.1885	11.8	29500	4.0783	0.3632
4.0772	12.0	30000	3.9658	0.3765
3.9602	12.2	30500	3.8686	0.3880
3.8622	12.4	31000	3.7886	0.3968
3.7958	12.6	31500	3.6968	0.4074
3.7245	12.8	32000	3.6480	0.4129
3.6503	13.0	32500	3.5771	0.4204
3.5569	13.2	33000	3.5103	0.4286
3.5151	13.4	33500	3.4611	0.4358
3.4388	13.6	34000	3.4119	0.4410
3.41	13.8	34500	3.3570	0.4486
3.3447	14.0	35000	3.3158	0.4518
3.2678	14.2	35500	3.2717	0.4585
3.2395	14.4	36000	3.2234	0.4629
3.2033	14.6	36500	3.1723	0.4697
3.1739	14.8	37000	3.1409	0.4747
3.1467	15.0	37500	3.1042	0.4782
3.0736	15.2	38000	3.0561	0.4839
3.0468	15.4	38500	3.0275	0.4869
3.0105	15.6	39000	3.0051	0.4898
2.9828	15.8	39500	2.9689	0.4950
2.9523	16.0	40000	2.9481	0.4959
2.8951	16.2	40500	2.8918	0.5039
2.8614	16.4	41000	2.8734	0.5054
2.8422	16.6	41500	2.8487	0.5083
2.8184	16.8	42000	2.8223	0.5138
2.7806	17.0	42500	2.7965	0.5167
2.7356	17.2	43000	2.7596	0.5209
2.7357	17.4	43500	2.7407	0.5250
2.7015	17.6	44000	2.7135	0.5272
2.688	17.8	44500	2.6935	0.5289
2.6582	18.0	45000	2.6572	0.5342
2.6186	18.2	45500	2.6396	0.5357
2.6071	18.4	46000	2.6270	0.5377
2.5891	18.6	46500	2.6110	0.5407
2.558	18.8	47000	2.5874	0.5435
2.5521	19.0	47500	2.5540	0.5465
2.5086	19.2	48000	2.5296	0.5504
2.4933	19.4	48500	2.5199	0.5523
2.4924	19.6	49000	2.5037	0.5550
2.4633	19.8	49500	2.4792	0.5567
2.4426	20.0	50000	2.4724	0.5600
2.4106	20.2	50500	2.4396	0.5626
2.4103	20.4	51000	2.4259	0.5631
2.3783	20.6	51500	2.4072	0.5672
2.3712	20.8	52000	2.4055	0.5679
2.3616	21.0	52500	2.3781	0.5724
2.3274	21.2	53000	2.3627	0.5746
2.3133	21.4	53500	2.3586	0.5751
2.3076	21.6	54000	2.3207	0.5785
2.2991	21.8	54500	2.3152	0.5796
2.2831	22.0	55000	2.3001	0.5815
2.2461	22.2	55500	2.2944	0.5822
2.2467	22.4	56000	2.2849	0.5856
2.2199	22.6	56500	2.2776	0.5863
2.2279	22.8	57000	2.2577	0.5885
2.2048	23.0	57500	2.2566	0.5886
2.1704	23.2	58000	2.2453	0.5914
2.1682	23.4	58500	2.2314	0.5927
2.1592	23.6	59000	2.2097	0.5961
2.1547	23.8	59500	2.1984	0.5972
2.1558	24.0	60000	2.1866	0.5993
2.1189	24.2	60500	2.1675	0.6009
2.1088	24.4	61000	2.1613	0.6028
2.1164	24.6	61500	2.1531	0.6046
2.094	24.8	62000	2.1507	0.6041
2.0977	25.0	62500	2.1299	0.6063
2.0657	25.2	63000	2.1218	0.6071
2.051	25.4	63500	2.1233	0.6083
2.0482	25.6	64000	2.1069	0.6100
2.04	25.8	64500	2.0985	0.6120
2.0341	26.0	65000	2.0929	0.6128
2.0207	26.2	65500	2.0767	0.6151
2.0044	26.4	66000	2.0672	0.6162
2.0037	26.6	66500	2.0623	0.6159
2.0081	26.8	67000	2.0614	0.6164
1.9847	27.0	67500	2.0499	0.6186
1.9465	27.2	68000	2.0399	0.6200
1.9573	27.4	68500	2.0353	0.6210
1.9682	27.6	69000	2.0187	0.6227
1.9573	27.8	69500	2.0251	0.6229
1.9491	28.0	70000	2.0086	0.6245
1.903	28.2	70500	2.0067	0.6246
1.9152	28.4	71000	1.9929	0.6264
1.9188	28.6	71500	1.9857	0.6274
1.9232	28.8	72000	1.9796	0.6287
1.9011	29.0	72500	1.9791	0.6289
1.8733	29.2	73000	1.9700	0.6289
1.8731	29.4	73500	1.9584	0.6303
1.8812	29.6	74000	1.9573	0.6323
1.8674	29.8	74500	1.9501	0.6318
1.8572	30.0	75000	1.9454	0.6333
1.849	30.2	75500	1.9375	0.6352
1.8332	30.4	76000	1.9344	0.6343
1.8413	30.6	76500	1.9293	0.6340
1.8298	30.8	77000	1.9228	0.6371
1.8336	31.0	77500	1.9215	0.6372
1.8122	31.2	78000	1.9133	0.6387
1.8001	31.4	78500	1.9119	0.6383
1.7934	31.6	79000	1.9088	0.6387
1.8079	31.8	79500	1.8940	0.6417
1.8017	32.0	80000	1.8889	0.6410
1.7789	32.2	80500	1.8883	0.6423
1.7739	32.4	81000	1.8836	0.6419
1.7602	32.6	81500	1.8795	0.6433
1.7731	32.8	82000	1.8769	0.6439
1.7784	33.0	82500	1.8590	0.6467
1.7506	33.2	83000	1.8664	0.6447
1.7307	33.4	83500	1.8553	0.6472
1.748	33.6	84000	1.8523	0.6470
1.7285	33.8	84500	1.8397	0.6491
1.7426	34.0	85000	1.8321	0.6492
1.7128	34.2	85500	1.8220	0.6507
1.7155	34.4	86000	1.8487	0.6479
1.7143	34.6	86500	1.8267	0.6504
1.7197	34.8	87000	1.8368	0.6499
1.7043	35.0	87500	1.8128	0.6524
1.6931	35.2	88000	1.8212	0.6517
1.6873	35.4	88500	1.8110	0.6531
1.684	35.6	89000	1.8145	0.6529
1.6802	35.8	89500	1.8046	0.6537
1.6807	36.0	90000	1.8016	0.6550
1.6612	36.2	90500	1.7997	0.6539
1.6586	36.4	91000	1.8014	0.6537
1.658	36.6	91500	1.7938	0.6565
1.6623	36.8	92000	1.7776	0.6586
1.6618	37.0	92500	1.7884	0.6573
1.6453	37.2	93000	1.7871	0.6571
1.6462	37.4	93500	1.7781	0.6585
1.6353	37.6	94000	1.7808	0.6583
1.6507	37.8	94500	1.7666	0.6603
1.6383	38.0	95000	1.7624	0.6606
1.6299	38.2	95500	1.7653	0.6605
1.6085	38.4	96000	1.7523	0.6610
1.6155	38.6	96500	1.7521	0.6612
1.6106	38.8	97000	1.7634	0.6605
1.6201	39.0	97500	1.7461	0.6625
1.5835	39.2	98000	1.7505	0.6617
1.5885	39.4	98500	1.7477	0.6623
1.5988	39.6	99000	1.7445	0.6635
1.6013	39.8	99500	1.7407	0.6637
1.594	40.0	100000	1.7336	0.6656
1.5741	40.2	100500	1.7348	0.6637
1.5744	40.4	101000	1.7242	0.6653
1.5809	40.6	101500	1.7262	0.6661
1.5723	40.8	102000	1.7257	0.6665
1.5695	41.0	102500	1.7182	0.6664
1.5462	41.2	103000	1.7257	0.6660
1.5545	41.4	103500	1.7101	0.6686
1.5574	41.6	104000	1.7108	0.6684
1.5485	41.8	104500	1.7164	0.6665
1.5487	42.0	105000	1.7080	0.6694
1.5278	42.2	105500	1.7092	0.6686
1.5282	42.4	106000	1.7052	0.6690
1.5468	42.6	106500	1.7058	0.6704
1.5375	42.8	107000	1.7020	0.6689
1.5301	43.0	107500	1.6950	0.6710
1.5224	43.2	108000	1.6990	0.6702
1.5105	43.4	108500	1.6919	0.6715
1.5179	43.6	109000	1.6845	0.6724
1.518	43.8	109500	1.6838	0.6721
1.5191	44.0	110000	1.6877	0.6715
1.4984	44.2	110500	1.6923	0.6712
1.5051	44.4	111000	1.6842	0.6722
1.4993	44.6	111500	1.6768	0.6741
1.5035	44.8	112000	1.6817	0.6727
1.5047	45.0	112500	1.6728	0.6733
1.4788	45.2	113000	1.6825	0.6720
1.4841	45.4	113500	1.6770	0.6735
1.4863	45.6	114000	1.6588	0.6753
1.4859	45.8	114500	1.6681	0.6741
1.4839	46.0	115000	1.6658	0.6740
1.4633	46.2	115500	1.6601	0.6765
1.4725	46.4	116000	1.6587	0.6753
1.4703	46.6	116500	1.6643	0.6756
1.4763	46.8	117000	1.6583	0.6759
1.4825	47.0	117500	1.6488	0.6766
1.4496	47.2	118000	1.6490	0.6772
1.457	47.4	118500	1.6462	0.6778
1.4541	47.6	119000	1.6368	0.6799
1.4561	47.8	119500	1.6404	0.6778
1.4547	48.0	120000	1.6385	0.6790
1.4406	48.2	120500	1.6374	0.6799
1.4374	48.4	121000	1.6319	0.6799
1.4395	48.6	121500	1.6425	0.6787
1.4347	48.8	122000	1.6252	0.6814
1.4392	49.0	122500	1.6360	0.6801
1.439	49.2	123000	1.6233	0.6826
1.4223	49.4	123500	1.6262	0.6809
1.4292	49.6	124000	1.6292	0.6811
1.4237	49.8	124500	1.6227	0.6812
1.4241	50.0	125000	1.6230	0.6810
1.4118	50.2	125500	1.6256	0.6822
1.4225	50.4	126000	1.6251	0.6817
1.4122	50.6	126500	1.6178	0.6827
1.4081	50.8	127000	1.6190	0.6813
1.4058	51.0	127500	1.6183	0.6836
1.3985	51.2	128000	1.6199	0.6817
1.3967	51.4	128500	1.6168	0.6829
1.4113	51.6	129000	1.6123	0.6832
1.3876	51.8	129500	1.6078	0.6841
1.4027	52.0	130000	1.6028	0.6847
1.3939	52.2	130500	1.6081	0.6845
1.391	52.4	131000	1.6034	0.6849
1.3895	52.6	131500	1.6016	0.6850
1.3858	52.8	132000	1.6010	0.6847
1.3852	53.0	132500	1.5886	0.6862
1.3716	53.2	133000	1.5964	0.6862
1.3727	53.4	133500	1.5952	0.6875
1.3656	53.6	134000	1.6031	0.6850
1.3873	53.8	134500	1.5927	0.6867
1.3742	54.0	135000	1.5970	0.6858
1.3687	54.2	135500	1.5954	0.6863
1.359	54.4	136000	1.5854	0.6873
1.3696	54.6	136500	1.5902	0.6878
1.38	54.8	137000	1.5870	0.6871
1.3529	55.0	137500	1.5888	0.6879
1.3479	55.2	138000	1.5720	0.6889
1.3558	55.4	138500	1.5810	0.6877
1.3565	55.6	139000	1.5687	0.6909
1.351	55.8	139500	1.5762	0.6897
1.3698	56.0	140000	1.5785	0.6881
1.3388	56.2	140500	1.5767	0.6882
1.3433	56.4	141000	1.5752	0.6896
1.3505	56.6	141500	1.5754	0.6890
1.3429	56.8	142000	1.5772	0.6896
1.337	57.0	142500	1.5732	0.6900
1.3398	57.2	143000	1.5681	0.6904
1.3334	57.4	143500	1.5696	0.6900
1.3384	57.6	144000	1.5674	0.6908
1.33	57.8	144500	1.5592	0.6916
1.327	58.0	145000	1.5498	0.6924
1.3234	58.2	145500	1.5626	0.6910
1.3266	58.4	146000	1.5743	0.6893
1.3152	58.6	146500	1.5680	0.6912
1.3279	58.8	147000	1.5581	0.6919
1.3172	59.0	147500	1.5645	0.6917
1.3073	59.2	148000	1.5579	0.6924
1.307	59.4	148500	1.5468	0.6939
1.3164	59.6	149000	1.5519	0.6930
1.3037	59.8	149500	1.5628	0.6917
1.3171	60.0	150000	1.5489	0.6934
1.3035	60.2	150500	1.5499	0.6931
1.3109	60.4	151000	1.5608	0.6922
1.304	60.6	151500	1.5612	0.6915
1.3104	60.8	152000	1.5511	0.6933
1.3071	61.0	152500	1.5469	0.6935
1.2935	61.2	153000	1.5485	0.6942
1.2866	61.4	153500	1.5463	0.6940
1.2926	61.6	154000	1.5406	0.6956
1.3029	61.8	154500	1.5424	0.6945
1.2921	62.0	155000	1.5446	0.6944
1.2765	62.2	155500	1.5397	0.6953
1.275	62.4	156000	1.5469	0.6945
1.2909	62.6	156500	1.5427	0.6945
1.2869	62.8	157000	1.5388	0.6949
1.2883	63.0	157500	1.5375	0.6948
1.2673	63.2	158000	1.5423	0.6948
1.2754	63.4	158500	1.5360	0.6957
1.2772	63.6	159000	1.5331	0.6952
1.283	63.8	159500	1.5354	0.6955
1.2737	64.0	160000	1.5388	0.6961
1.2681	64.2	160500	1.5382	0.6952
1.2769	64.4	161000	1.5350	0.6958
1.2668	64.6	161500	1.5345	0.6956
1.2795	64.8	162000	1.5198	0.6984
1.2632	65.0	162500	1.5324	0.6965
1.2646	65.2	163000	1.5420	0.6963
1.2739	65.4	163500	1.5308	0.6972
1.2667	65.6	164000	1.5220	0.6978
1.26	65.8	164500	1.5283	0.6961
1.2714	66.0	165000	1.5235	0.6977
1.2652	66.2	165500	1.5270	0.6973
1.2554	66.4	166000	1.5356	0.6959
1.2666	66.6	166500	1.5231	0.6968
1.2634	66.8	167000	1.5170	0.6994
1.2485	67.0	167500	1.5205	0.6988
1.2397	67.2	168000	1.5263	0.6982
1.2416	67.4	168500	1.5133	0.7001
1.2615	67.6	169000	1.5150	0.6990
1.254	67.8	169500	1.5213	0.6990
1.2463	68.0	170000	1.5157	0.6995
1.2412	68.2	170500	1.5082	0.7000
1.248	68.4	171000	1.5168	0.6992
1.2468	68.6	171500	1.5187	0.6991
1.246	68.8	172000	1.5094	0.7001
1.2443	69.0	172500	1.5186	0.6984
1.2451	69.2	173000	1.5157	0.6988
1.2375	69.4	173500	1.5102	0.7002
1.2441	69.6	174000	1.5143	0.7000
1.2335	69.8	174500	1.5173	0.6985
1.2361	70.0	175000	1.5102	0.7001
1.23	70.2	175500	1.5155	0.6997
1.2401	70.4	176000	1.5027	0.7005
1.2346	70.6	176500	1.5123	0.6995
1.2306	70.8	177000	1.5151	0.6984
1.2333	71.0	177500	1.5125	0.7000
1.2248	71.2	178000	1.5199	0.6992
1.2385	71.4	178500	1.5108	0.7000
1.2278	71.6	179000	1.5092	0.7000
1.2278	71.8	179500	1.5163	0.6989
1.2242	72.0	180000	1.5056	0.7010
1.2208	72.2	180500	1.4968	0.7023
1.2216	72.4	181000	1.5097	0.7006
1.2271	72.6	181500	1.4988	0.7013
1.2302	72.8	182000	1.5141	0.6997
1.2268	73.0	182500	1.4996	0.7016
1.2258	73.2	183000	1.5016	0.7008
1.2244	73.4	183500	1.5032	0.7012
1.2117	73.6	184000	1.5097	0.7003
1.2279	73.8	184500	1.5058	0.7012
1.2274	74.0	185000	1.5030	0.7015
1.2117	74.2	185500	1.5086	0.7008
1.2223	74.4	186000	1.4998	0.7018
1.227	74.6	186500	1.5059	0.7014
1.2168	74.8	187000	1.4984	0.7011
1.2093	75.0	187500	1.5077	0.7018

Framework versions

• Transformers 4.25.1
• Pytorch 1.13.1+cu117
• Datasets 2.11.0
• Tokenizers 0.13.2