clinical_transcripts_roberta

This model is a fine-tuned version of allenai/biomed_roberta_base on medical transcriptions dataset. It achieves the following results on the evaluation set:

• Loss: 1.0331

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• lr_scheduler_warmup_steps: 100
• training_steps: 4000

Training results

Training Loss	Epoch	Step	Validation Loss
1.405	0.51	100	1.2925
1.316	1.01	200	1.2107
1.2781	1.52	300	1.1704
1.2911	2.02	400	1.1745
1.2241	2.53	500	1.1730
1.2063	3.03	600	1.1248
1.174	3.54	700	1.1416
1.1588	4.04	800	1.1495
1.1513	4.55	900	1.1145
1.1541	5.05	1000	1.1402
1.1266	5.56	1100	1.1156
1.1205	6.06	1200	1.1075
1.1141	6.57	1300	1.1157
1.0956	7.07	1400	1.1047
1.0809	7.58	1500	1.0921
1.0755	8.08	1600	1.0891
1.044	8.59	1700	1.0758
1.1103	9.09	1800	1.0881
1.0578	9.6	1900	1.0578
1.0462	10.1	2000	1.1043
1.0302	10.61	2100	1.0787
1.0236	11.11	2200	1.0841
1.0371	11.62	2300	1.0904
1.0178	12.12	2400	1.0593
0.999	12.63	2500	1.0661
0.9867	13.13	2600	1.0670
0.9986	13.64	2700	1.0470
0.9867	14.14	2800	1.0347
0.9848	14.65	2900	1.0274
0.9627	15.15	3000	1.0550
0.9659	15.66	3100	1.0499
0.9743	16.16	3200	1.0419
0.9507	16.67	3300	1.0679
0.941	17.17	3400	1.0142
0.9548	17.68	3500	1.0422
0.9378	18.18	3600	1.0471
0.9339	18.69	3700	1.0473
0.9195	19.19	3800	1.0248
0.9254	19.7	3900	1.0235
0.9393	20.2	4000	1.0331

Framework versions

• Transformers 4.34.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.5
• Tokenizers 0.14.0