Hugging Face's logo

lyan62
/
zh_wiki_small

Transformers
PyTorch
pixel
masked-auto-encoding
Generated from Trainer
Inference Endpoints
Model card Files Community
zh_wiki_small / README.md
lyan62's picture
lyan62
update model card README.md
0631e18
preview code
|
raw
history blame
27.9 kB
metadata
tags:
  - masked-auto-encoding
  - generated_from_trainer
model-index:
  - name: zh_wiki_small
    results: []

zh_wiki_small

This model is a fine-tuned version of on the wikipedia dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4159

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.00015
  • train_batch_size: 32
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 256
  • total_eval_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.05
  • training_steps: 500000
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
0.4793 0.14 1000 0.4540
0.4701 0.28 2000 0.4467
0.4673 0.42 3000 0.4304
0.4669 0.55 4000 0.4413
0.4668 0.69 5000 0.4368
0.4676 0.83 6000 0.4358
0.4691 0.97 7000 0.4367
0.4693 1.11 8000 0.4429
0.4709 1.25 9000 0.4388
0.4722 1.39 10000 0.4453
0.4729 1.53 11000 0.4415
0.4732 1.67 12000 0.4510
0.4751 1.8 13000 0.4461
0.4765 1.94 14000 0.4448
0.477 2.08 15000 0.4498
0.4779 2.22 16000 0.4447
0.4795 2.36 17000 0.4430
0.481 2.5 18000 0.4499
0.4821 2.64 19000 0.4551
0.4829 2.78 20000 0.4519
0.4838 2.91 21000 0.4520
0.4856 3.05 22000 0.4633
0.4857 3.19 23000 0.4576
0.4869 3.33 24000 0.4485
0.4882 3.47 25000 0.4591
0.4883 3.61 26000 0.4645
0.4889 3.75 27000 0.4570
0.4884 3.89 28000 0.4572
0.4897 4.02 29000 0.4553
0.4883 4.16 30000 0.4534
0.4881 4.3 31000 0.4587
0.4889 4.44 32000 0.4632
0.4886 4.58 33000 0.4587
0.4883 4.72 34000 0.4621
0.4876 4.86 35000 0.4522
0.4878 5.0 36000 0.4560
0.4883 5.13 37000 0.4579
0.4882 5.27 38000 0.4554
0.4883 5.41 39000 0.4588
0.4872 5.55 40000 0.4561
0.4868 5.69 41000 0.4614
0.4875 5.83 42000 0.4584
0.4868 5.97 43000 0.4619
0.4874 6.11 44000 0.4519
0.4874 6.24 45000 0.4625
0.487 6.38 46000 0.4579
0.4872 6.52 47000 0.4534
0.4872 6.66 48000 0.4516
0.4865 6.8 49000 0.4635
0.4865 6.94 50000 0.4610
0.4863 7.08 51000 0.4515
0.4861 7.22 52000 0.4584
0.4866 7.35 53000 0.4541
0.4862 7.49 54000 0.4508
0.4863 7.63 55000 0.4565
0.486 7.77 56000 0.4665
0.486 7.91 57000 0.4565
0.4861 8.05 58000 0.4580
0.4852 8.19 59000 0.4596
0.4846 8.33 60000 0.4527
0.4848 8.46 61000 0.4505
0.4849 8.6 62000 0.4407
0.4851 8.74 63000 0.4579
0.4848 8.88 64000 0.4559
0.4851 9.02 65000 0.4505
0.4846 9.16 66000 0.4615
0.4842 9.3 67000 0.4618
0.484 9.44 68000 0.4559
0.4841 9.57 69000 0.4613
0.484 9.71 70000 0.4527
0.4842 9.85 71000 0.4483
0.4842 9.99 72000 0.4585
0.4837 10.13 73000 0.4585
0.4833 10.27 74000 0.4541
0.4836 10.41 75000 0.4528
0.4832 10.55 76000 0.4475
0.4836 10.68 77000 0.4525
0.4826 10.82 78000 0.4562
0.4824 10.96 79000 0.4502
0.4828 11.1 80000 0.4529
0.4829 11.24 81000 0.4524
0.4823 11.38 82000 0.4506
0.4827 11.52 83000 0.4511
0.4823 11.66 84000 0.4506
0.4827 11.79 85000 0.4561
0.4832 11.93 86000 0.4471
0.482 12.07 87000 0.4479
0.4819 12.21 88000 0.4561
0.4816 12.35 89000 0.4590
0.4818 12.49 90000 0.4469
0.4815 12.63 91000 0.4633
0.4822 12.77 92000 0.4566
0.4816 12.9 93000 0.4548
0.4824 13.04 94000 0.4548
0.4812 13.18 95000 0.4533
0.4809 13.32 96000 0.4546
0.481 13.46 97000 0.4590
0.4807 13.6 98000 0.4465
0.4808 13.74 99000 0.4531
0.4806 13.88 100000 0.4459
0.4809 14.01 101000 0.4517
0.4801 14.15 102000 0.4519
0.4801 14.29 103000 0.4547
0.4805 14.43 104000 0.4517
0.4799 14.57 105000 0.4491
0.4805 14.71 106000 0.4559
0.48 14.85 107000 0.4551
0.4796 14.99 108000 0.4537
0.4801 15.12 109000 0.4509
0.4797 15.26 110000 0.4482
0.4798 15.4 111000 0.4466
0.4789 15.54 112000 0.4445
0.4808 15.68 113000 0.4493
0.4789 15.82 114000 0.4475
0.4792 15.96 115000 0.4543
0.4787 16.1 116000 0.4471
0.4796 16.23 117000 0.4565
0.4787 16.37 118000 0.4515
0.4788 16.51 119000 0.4449
0.4783 16.65 120000 0.4454
0.4787 16.79 121000 0.4486
0.4789 16.93 122000 0.4480
0.4782 17.07 123000 0.4529
0.4782 17.21 124000 0.4481
0.4777 17.34 125000 0.4528
0.4779 17.48 126000 0.4514
0.4781 17.62 127000 0.4520
0.4776 17.76 128000 0.4495
0.4777 17.9 129000 0.4501
0.4783 18.04 130000 0.4528
0.4771 18.18 131000 0.4498
0.4775 18.32 132000 0.4525
0.4772 18.45 133000 0.4482
0.4775 18.59 134000 0.4532
0.4769 18.73 135000 0.4537
0.4776 18.87 136000 0.4509
0.4775 19.01 137000 0.4464
0.4769 19.15 138000 0.4464
0.4772 19.29 139000 0.4499
0.4766 19.43 140000 0.4428
0.4764 19.56 141000 0.4536
0.477 19.7 142000 0.4444
0.4764 19.84 143000 0.4482
0.4764 19.98 144000 0.4510
0.4763 20.12 145000 0.4519
0.4761 20.26 146000 0.4452
0.4761 20.4 147000 0.4476
0.4756 20.54 148000 0.4494
0.4757 20.67 149000 0.4544
0.4762 20.81 150000 0.4412
0.4757 20.95 151000 0.4459
0.4749 21.09 152000 0.4532
0.4752 21.23 153000 0.4477
0.4749 21.37 154000 0.4396
0.4764 21.51 155000 0.4466
0.4753 21.65 156000 0.4523
0.4755 21.78 157000 0.4582
0.4749 21.92 158000 0.4539
0.475 22.06 159000 0.4539
0.4747 22.2 160000 0.4519
0.4745 22.34 161000 0.4370
0.4748 22.48 162000 0.4449
0.4743 22.62 163000 0.4484
0.4745 22.76 164000 0.4471
0.4739 22.89 165000 0.4480
0.4746 23.03 166000 0.4519
0.4739 23.17 167000 0.4478
0.4739 23.31 168000 0.4497
0.4738 23.45 169000 0.4462
0.474 23.59 170000 0.4430
0.4737 23.73 171000 0.4483
0.4737 23.87 172000 0.4508
0.474 24.0 173000 0.4439
0.4729 24.14 174000 0.4426
0.4735 24.28 175000 0.4433
0.4722 24.42 176000 0.4483
0.4728 24.56 177000 0.4496
0.4727 24.7 178000 0.4473
0.4729 24.84 179000 0.4404
0.4722 24.98 180000 0.4426
0.4724 25.11 181000 0.4479
0.4739 25.25 182000 0.4430
0.4723 25.39 183000 0.4418
0.4724 25.53 184000 0.4371
0.472 25.67 185000 0.4456
0.4726 25.81 186000 0.4419
0.4721 25.95 187000 0.4417
0.4722 26.09 188000 0.4475
0.4715 26.22 189000 0.4389
0.4717 26.36 190000 0.4451
0.4716 26.5 191000 0.4440
0.4714 26.64 192000 0.4399
0.4712 26.78 193000 0.4398
0.4709 26.92 194000 0.4424
0.4714 27.06 195000 0.4533
0.4706 27.2 196000 0.4394
0.471 27.33 197000 0.4436
0.4707 27.47 198000 0.4421
0.471 27.61 199000 0.4459
0.4707 27.75 200000 0.4439
0.471 27.89 201000 0.4467
0.471 28.03 202000 0.4439
0.4704 28.17 203000 0.4445
0.4705 28.31 204000 0.4429
0.4706 28.44 205000 0.4382
0.4703 28.58 206000 0.4425
0.4695 28.72 207000 0.4414
0.4696 28.86 208000 0.4405
0.4696 29.0 209000 0.4460
0.4701 29.14 210000 0.4460
0.4696 29.28 211000 0.4397
0.4693 29.42 212000 0.4439
0.4694 29.55 213000 0.4495
0.469 29.69 214000 0.4466
0.4691 29.83 215000 0.4336
0.4694 29.97 216000 0.4377
0.4698 30.11 217000 0.4356
0.4689 30.25 218000 0.4381
0.4685 30.39 219000 0.4431
0.4688 30.53 220000 0.4411
0.4687 30.66 221000 0.4445
0.4685 30.8 222000 0.4432
0.4687 30.94 223000 0.4383
0.4681 31.08 224000 0.4371
0.4683 31.22 225000 0.4384
0.4678 31.36 226000 0.4396
0.4682 31.5 227000 0.4387
0.4671 31.64 228000 0.4382
0.4676 31.77 229000 0.4410
0.4681 31.91 230000 0.4391
0.4676 32.05 231000 0.4429
0.4673 32.19 232000 0.4395
0.4669 32.33 233000 0.4389
0.4675 32.47 234000 0.4452
0.4667 32.61 235000 0.4395
0.4667 32.75 236000 0.4460
0.4672 32.88 237000 0.4404
0.4667 33.02 238000 0.4372
0.4663 33.16 239000 0.4362
0.4669 33.3 240000 0.4428
0.4662 33.44 241000 0.4370
0.4662 33.58 242000 0.4382
0.466 33.72 243000 0.4395
0.4661 33.86 244000 0.4418
0.4663 33.99 245000 0.4407
0.4661 34.13 246000 0.4346
0.4652 34.27 247000 0.4392
0.4662 34.41 248000 0.4396
0.4655 34.55 249000 0.4427
0.4657 34.69 250000 0.4484
0.4654 34.83 251000 0.4268
0.4655 34.97 252000 0.4384
0.4649 35.1 253000 0.4383
0.465 35.24 254000 0.4368
0.4648 35.38 255000 0.4327
0.4647 35.52 256000 0.4416
0.4652 35.66 257000 0.4390
0.4646 35.8 258000 0.4450
0.4651 35.94 259000 0.4354
0.4643 36.08 260000 0.4473
0.464 36.21 261000 0.4423
0.4638 36.35 262000 0.4339
0.464 36.49 263000 0.4438
0.464 36.63 264000 0.4398
0.4637 36.77 265000 0.4352
0.4641 36.91 266000 0.4352
0.4651 37.05 267000 0.4324
0.4637 37.19 268000 0.4341
0.4633 37.32 269000 0.4331
0.4639 37.46 270000 0.4391
0.463 37.6 271000 0.4380
0.4635 37.74 272000 0.4355
0.4631 37.88 273000 0.4397
0.464 38.02 274000 0.4336
0.4629 38.16 275000 0.4339
0.4634 38.3 276000 0.4355
0.4632 38.43 277000 0.4388
0.4628 38.57 278000 0.4341
0.4621 38.71 279000 0.4337
0.4626 38.85 280000 0.4340
0.462 38.99 281000 0.4306
0.8286 39.13 282000 0.4504
0.4624 39.27 283000 0.4399
0.4621 39.41 284000 0.4351
0.4622 39.54 285000 0.4304
0.4619 39.68 286000 0.4329
0.4618 39.82 287000 0.4208
0.462 39.96 288000 0.4414
0.4615 40.1 289000 0.4353
0.4614 40.24 290000 0.4398
0.4611 40.38 291000 0.4371
0.4608 40.52 292000 0.4326
0.4611 40.65 293000 0.4332
0.4614 40.79 294000 0.4343
0.4609 40.93 295000 0.4306
0.4608 41.07 296000 0.4323
0.4608 41.21 297000 0.4321
0.4601 41.35 298000 0.4330
0.4606 41.49 299000 0.4361
0.4606 41.63 300000 0.4367
0.46 41.76 301000 0.4327
0.4596 41.9 302000 0.4306
0.46 42.04 303000 0.4352
0.46 42.18 304000 0.4338
0.4597 42.32 305000 0.4333
0.4596 42.46 306000 0.4334
0.4591 42.6 307000 0.4334
0.4597 42.74 308000 0.4319
0.4586 42.87 309000 0.4268
0.4593 43.01 310000 0.4366
0.4591 43.15 311000 0.4283
0.4587 43.29 312000 0.4289
0.4594 43.43 313000 0.4332
0.459 43.57 314000 0.4326
0.4586 43.71 315000 0.4356
0.4581 43.85 316000 0.4271
0.4584 43.98 317000 0.4325
0.4586 44.12 318000 0.4350
0.4584 44.26 319000 0.4273
0.4576 44.4 320000 0.4284
0.458 44.54 321000 0.4331
0.4581 44.68 322000 0.4263
0.4579 44.82 323000 0.4283
0.4583 44.96 324000 0.4362
0.4571 45.1 325000 0.4330
0.4566 45.23 326000 0.4300
0.4572 45.37 327000 0.4258
0.4574 45.51 328000 0.4200
0.4573 45.65 329000 0.4299
0.4578 45.79 330000 0.4319
0.4576 45.93 331000 0.4352
0.4574 46.07 332000 0.4278
0.4572 46.21 333000 0.4326
0.4568 46.34 334000 0.4295
0.4569 46.48 335000 0.4300
0.4566 46.62 336000 0.4333
0.4567 46.76 337000 0.4262
0.4564 46.9 338000 0.4354
0.4574 47.04 339000 0.4357
0.4564 47.18 340000 0.4308
0.4554 47.32 341000 0.4350
0.456 47.45 342000 0.4400
0.456 47.59 343000 0.4237
0.4559 47.73 344000 0.4236
0.4559 47.87 345000 0.4305
0.4559 48.01 346000 0.4245
0.4549 48.15 347000 0.4182
0.4556 48.29 348000 0.4330
0.4551 48.43 349000 0.4397
0.455 48.56 350000 0.4252
0.4548 48.7 351000 0.4246
0.4551 48.84 352000 0.4291
0.4554 48.98 353000 0.4286
0.4547 49.12 354000 0.4336
0.4548 49.26 355000 0.4324
0.4545 49.4 356000 0.4236
0.4547 49.54 357000 0.4345
0.4542 49.67 358000 0.4329
0.4545 49.81 359000 0.4241
0.4541 49.95 360000 0.4177
0.454 50.09 361000 0.4244
0.4538 50.23 362000 0.4190
0.4535 50.37 363000 0.4331
0.4545 50.51 364000 0.4252
0.454 50.65 365000 0.4315
0.4536 50.78 366000 0.4301
0.4534 50.92 367000 0.4357
0.4537 51.06 368000 0.4334
0.4535 51.2 369000 0.4200
0.4538 51.34 370000 0.4274
0.4536 51.48 371000 0.4178
0.4534 51.62 372000 0.4181
0.4533 51.76 373000 0.4211
0.4535 51.89 374000 0.4290
0.4535 52.03 375000 0.4201
0.4526 52.17 376000 0.4263
0.4526 52.31 377000 0.4237
0.4524 52.45 378000 0.4254
0.4529 52.59 379000 0.4260
0.4531 52.73 380000 0.4202
0.4523 52.87 381000 0.4223
0.4523 53.0 382000 0.4271
0.4522 53.14 383000 0.4286
0.4524 53.28 384000 0.4256
0.4515 53.42 385000 0.4221
0.4513 53.56 386000 0.4255
0.452 53.7 387000 0.4270
0.4519 53.84 388000 0.4222
0.4518 53.98 389000 0.4233
0.4513 54.11 390000 0.4233
0.4517 54.25 391000 0.4239
0.4518 54.39 392000 0.4273
0.4508 54.53 393000 0.4200
0.4511 54.67 394000 0.4236
0.4508 54.81 395000 0.4193
0.4507 54.95 396000 0.4293
0.4508 55.09 397000 0.4187
0.4504 55.22 398000 0.4283
0.4512 55.36 399000 0.4239
0.4504 55.5 400000 0.4269
0.4506 55.64 401000 0.4291
0.4504 55.78 402000 0.4238
0.4503 55.92 403000 0.4200
0.4506 56.06 404000 0.4186
0.4507 56.2 405000 0.4260
0.4504 56.33 406000 0.4188
0.4503 56.47 407000 0.4231
0.4498 56.61 408000 0.4148
0.4499 56.75 409000 0.4182
0.4498 56.89 410000 0.4229
0.4501 57.03 411000 0.4252
0.4497 57.17 412000 0.4220
0.45 57.31 413000 0.4181
0.4497 57.44 414000 0.4270
0.4497 57.58 415000 0.4208
0.4499 57.72 416000 0.4224
0.4496 57.86 417000 0.4207
0.4494 58.0 418000 0.4268
0.4499 58.14 419000 0.4240
0.4495 58.28 420000 0.4294
0.4487 58.42 421000 0.4207
0.4495 58.55 422000 0.4246
0.4491 58.69 423000 0.4213
0.4492 58.83 424000 0.4241
0.4486 58.97 425000 0.4247
0.4485 59.11 426000 0.4163
0.4489 59.25 427000 0.4239
0.4483 59.39 428000 0.4240
0.4491 59.53 429000 0.4214
0.4485 59.66 430000 0.4285
0.449 59.8 431000 0.4265
0.4484 59.94 432000 0.4188
0.4484 60.08 433000 0.4176
0.4488 60.22 434000 0.4200
0.448 60.36 435000 0.4116
0.4477 60.5 436000 0.4215
0.4484 60.64 437000 0.4204
0.448 60.77 438000 0.4093
0.4479 60.91 439000 0.4181
0.4481 61.05 440000 0.4232
0.4477 61.19 441000 0.4202
0.4478 61.33 442000 0.4167
0.4481 61.47 443000 0.4173
0.4483 61.61 444000 0.4158
0.4473 61.75 445000 0.4174
0.4474 61.88 446000 0.4266
0.4477 62.02 447000 0.4242
0.4476 62.16 448000 0.4240
0.4478 62.3 449000 0.4286
0.4474 62.44 450000 0.4294
0.4482 62.58 451000 0.4144
0.4471 62.72 452000 0.4316
0.448 62.86 453000 0.4228
0.4474 62.99 454000 0.4242
0.447 63.13 455000 0.4231
0.4475 63.27 456000 0.4235
0.4475 63.41 457000 0.4279
0.4476 63.55 458000 0.4230
0.4464 63.69 459000 0.4145
0.4467 63.83 460000 0.4230
0.4465 63.97 461000 0.4208
0.4466 64.1 462000 0.4243
0.447 64.24 463000 0.4220
0.4473 64.38 464000 0.4253
0.4471 64.52 465000 0.4194
0.447 64.66 466000 0.4262
0.447 64.8 467000 0.4245
0.4468 64.94 468000 0.4143
0.4463 65.08 469000 0.4187
0.4465 65.21 470000 0.4185
0.4465 65.35 471000 0.4244
0.4467 65.49 472000 0.4201
0.4465 65.63 473000 0.4160
0.4467 65.77 474000 0.4273
0.4465 65.91 475000 0.4183
0.4467 66.05 476000 0.4227
0.4469 66.19 477000 0.4166
0.4467 66.32 478000 0.4199
0.4464 66.46 479000 0.4181
0.4463 66.6 480000 0.4217
0.4464 66.74 481000 0.4158
0.4468 66.88 482000 0.4191
0.447 67.02 483000 0.4248
0.4465 67.16 484000 0.4234
0.4463 67.3 485000 0.4238
0.446 67.43 486000 0.4162
0.4462 67.57 487000 0.4202
0.4462 67.71 488000 0.4177
0.4455 67.85 489000 0.4228
0.4463 67.99 490000 0.4146
0.4454 68.13 491000 0.4190
0.446 68.27 492000 0.4219
0.4461 68.41 493000 0.4250
0.4462 68.54 494000 0.4172
0.4464 68.68 495000 0.4122
0.4459 68.82 496000 0.4178
0.4459 68.96 497000 0.4095
0.4458 69.1 498000 0.4124
0.4458 69.24 499000 0.4182
0.4458 69.38 500000 0.4177

Framework versions

  • Transformers 4.17.0
  • Pytorch 1.12.0
  • Datasets 2.0.0
  • Tokenizers 0.13.2