bart-cnn-pubhealth-expanded-hi-grad

This model is a fine-tuned version of facebook/bart-large-cnn on the clupubhealth dataset. It achieves the following results on the evaluation set:

• Loss: 2.1939
• Rouge1: 28.8807
• Rouge2: 8.9567
• Rougel: 19.5591
• Rougelsum: 20.6726
• Gen Len: 66.99

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 950
• total_train_batch_size: 15200
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
3.4166	0.49	2	2.4019	22.0991	4.6789	15.1628	17.4382	75.065
3.2194	0.98	4	2.3372	25.0981	6.6975	17.4606	19.2018	71.225
3.0969	1.47	6	2.2979	26.4747	7.1948	18.2262	19.6241	67.19
3.0313	1.96	8	2.3038	26.8637	7.5831	18.2923	19.6327	66.875
2.9753	2.44	10	2.2976	27.8942	8.3434	19.095	20.6248	67.975
2.9296	2.93	12	2.2602	28.1255	8.6477	19.0575	20.7787	68.515
2.8681	3.42	14	2.2341	28.0812	8.598	19.3391	20.7526	68.285
2.867	3.91	16	2.2246	28.3624	8.7921	19.5552	21.1147	68.225
2.8157	4.4	18	2.2178	28.8197	8.8423	19.3606	20.698	69.08
2.8007	4.89	20	2.2149	28.34	8.5084	18.8293	20.1169	68.255
2.7797	5.38	22	2.2123	28.2156	8.4891	19.3472	20.5036	67.525
2.7563	5.87	24	2.2083	27.8927	8.3783	19.1194	20.2498	68.365
2.736	6.36	26	2.2035	28.2588	8.2345	18.9418	20.2931	68.335
2.7208	6.85	28	2.2020	28.2471	8.599	19.3465	20.5104	68.44
2.713	7.33	30	2.2022	28.1863	8.5142	19.194	20.2467	68.3
2.7135	7.82	32	2.2013	28.462	8.6346	19.2465	20.4812	68.195
2.6987	8.31	34	2.1988	28.9168	8.8888	19.6491	20.7796	67.275
2.6978	8.8	36	2.1965	28.7303	8.9879	19.5924	20.6943	67.31
2.6769	9.29	38	2.1946	28.7956	8.9652	19.545	20.7352	67.33
2.6821	9.78	40	2.1939	28.8807	8.9567	19.5591	20.6726	66.99

Framework versions

• Transformers 4.31.0
• Pytorch 2.0.1+cu117
• Datasets 2.7.1
• Tokenizers 0.13.2