Camera ready

README.md

@@ -12,34 +12,46 @@ language:
 widget:
 - text: "Мы сели в тачку и поехали по ресторанам. Что такое тачка?"
   example_title: "Definition generation"
+license: cc-by-sa-4.0
 ---
 
-# mt0-definition-ru-xl
+# mT0-Definition-Ru XL
 
-This model is a version of [mt0-xl](https://huggingface.co/bigscience/mt0-xl) finetuned on the Russian part of CoDWoE dataset.
+This model is a version of [mT0 XL](https://huggingface.co/bigscience/mt0-xl) finetuned on the Russian part of [CodWoE](https://aclanthology.org/2022.semeval-1.1/),
+a dataset of definitions and usage examples.
 
-It achieves the following results on the evaluation set:
-- Loss: 1.6241
-- Rouge1: 0.2536
-- Rouge2: 0.003
-- Rougel: 0.2531
-- Rougelsum: 0.2527
-- Gen Len: 24.0693
+It generates definitions of Russian words in context.
+Its input is the usage example and the instruction question "Что такое TARGET_WORD?"
 
 ## Model description
 
-More information needed
+See details in the paper `Enriching Word Usage Graphs with Cluster Definitions` (LREC-COLING'2024) by
+Mariia Fedorova, Andrey Kutuzov, Nikolay Arefyev and Dominik Schlechtweg.
 
 ## Intended uses & limitations
 
-More information needed
+The model is intended for research purposes, as a source of contextualized dictionary-like lexical definitions.
+Generated definitions can contain all sorts of biases and stereotypes, stemming from the underlying language model.
 
 ## Training and evaluation data
 
-More information needed
+Russian subset of *CodWoE* ([Mickus et al., SemEval 2022](https://aclanthology.org/2022.semeval-1.1/)).
+
+## Training results
+
+mT0-Definition-Ru XL achieves the following results on the CodWoE evaluation set:
+
+- Loss: 1.7996
+- Rouge1: 17.5576
+- Rouge2: 8.7614
+- Rougel: 17.2533
+- Rougelsum: 17.3204
+- Gen Len: 21.6774
 
 ## Training procedure
 
+mT0-Definition-Ru XL was fine-tuned in a sequence-to-sequence mode on examples of contextualized dictionary definitions.
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -56,23 +68,11 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: linear
 - num_epochs: 20.0
 
-### Training results
-
-| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
-|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
-| 2.0449        | 1.0   | 512  | 1.6755          | 0.0817 | 0.0    | 0.0778 | 0.0817    | 16.7581 |
-| 1.707         | 2.0   | 1025 | 1.6182          | 0.096  | 0.0    | 0.097  | 0.1       | 15.8621 |
-| 1.5398        | 3.0   | 1537 | 1.6085          | 0.1394 | 0.0034 | 0.1401 | 0.1416    | 16.4765 |
-| 1.4142        | 4.0   | 2050 | 1.6016          | 0.1132 | 0.0    | 0.1132 | 0.1098    | 16.2732 |
-| 1.3102        | 5.0   | 2562 | 1.6241          | 0.2082 | 0.0034 | 0.2054 | 0.2061    | 16.2877 |
-| 1.2162        | 6.0   | 3075 | 1.6281          | 0.1549 | 0.0    | 0.1549 | 0.1549    | 16.1581 |
-| 1.1364        | 7.0   | 3587 | 1.6622          | 0.1583 | 0.0    | 0.1575 | 0.1589    | 15.9925 |
-| 1.0649        | 8.0   | 4100 | 1.6812          | 0.2033 | 0.0137 | 0.2012 | 0.2027    | 16.5099 |
-
-
 ### Framework versions
 
-- Transformers 4.30.2
+- Transformers 4.37.1
 - Pytorch 1.13.1+rocm5.2
-- Datasets 2.12.0
-- Tokenizers 0.12.1
+- Datasets 2.16.1
+- Tokenizers 0.15.1
+
+## Citation

all_results.json

@@ -1,18 +1,18 @@
 {
-    "epoch": 8.0,
-    "eval_gen_len": 24.06929198682766,
-    "eval_loss": 1.6240657567977905,
-    "eval_rouge1": 0.2536,
-    "eval_rouge2": 0.003,
-    "eval_rougeL": 0.2531,
-    "eval_rougeLsum": 0.2527,
-    "eval_runtime": 635.3216,
+    "epoch": 15.0,
+    "eval_gen_len": 21.67735745614035,
+    "eval_loss": 1.7995822429656982,
+    "eval_rouge1": 17.5576,
+    "eval_rouge2": 8.7614,
+    "eval_rougeL": 17.2533,
+    "eval_rougeLsum": 17.3204,
+    "eval_runtime": 423.3617,
     "eval_samples": 7288,
-    "eval_samples_per_second": 11.471,
-    "eval_steps_per_second": 0.359,
-    "train_loss": 1.429130665848895,
-    "train_runtime": 9768.8261,
+    "eval_samples_per_second": 17.215,
+    "eval_steps_per_second": 0.539,
+    "train_loss": 1.153788715837463,
+    "train_runtime": 20083.7121,
     "train_samples": 65584,
-    "train_samples_per_second": 134.272,
-    "train_steps_per_second": 1.048
+    "train_samples_per_second": 65.311,
+    "train_steps_per_second": 0.51
 }

config.json

@@ -3,6 +3,7 @@
   "architectures": [
     "MT5ForConditionalGeneration"
   ],
+  "classifier_dropout": 0.0,
   "d_ff": 5120,
   "d_kv": 64,
   "d_model": 2048,
@@ -26,7 +27,7 @@
   "tie_word_embeddings": false,
   "tokenizer_class": "T5Tokenizer",
   "torch_dtype": "float32",
-  "transformers_version": "4.30.2",
+  "transformers_version": "4.37.1",
   "use_cache": true,
   "vocab_size": 250112
 }

eval_results.json

@@ -1,13 +1,13 @@
 {
-    "epoch": 8.0,
-    "eval_gen_len": 24.06929198682766,
-    "eval_loss": 1.6240657567977905,
-    "eval_rouge1": 0.2536,
-    "eval_rouge2": 0.003,
-    "eval_rougeL": 0.2531,
-    "eval_rougeLsum": 0.2527,
-    "eval_runtime": 635.3216,
+    "epoch": 15.0,
+    "eval_gen_len": 21.67735745614035,
+    "eval_loss": 1.7995822429656982,
+    "eval_rouge1": 17.5576,
+    "eval_rouge2": 8.7614,
+    "eval_rougeL": 17.2533,
+    "eval_rougeLsum": 17.3204,
+    "eval_runtime": 423.3617,
     "eval_samples": 7288,
-    "eval_samples_per_second": 11.471,
-    "eval_steps_per_second": 0.359
+    "eval_samples_per_second": 17.215,
+    "eval_steps_per_second": 0.539
 }

generation_config.json

@@ -2,5 +2,5 @@
   "decoder_start_token_id": 0,
   "eos_token_id": 1,
   "pad_token_id": 0,
-  "transformers_version": "4.30.2"
+  "transformers_version": "4.37.1"
 }

pytorch_model-00001-of-00002.bin → pytorch_model-00001-of-00003.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b0519cc6dc79c0fcbc0f8f27d4e48c28178c7c577df247a331dc728c70182766
-size 9977020596
+oid sha256:883fe03c74638936701aa6db8ea888ebcdfcecaf9a5e896ed5d0db65dbefa436
+size 4993619647

pytorch_model-00002-of-00003.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06c1a331261c210d7adb2977052855975e876fcf786ac4df398d3a638c6af82c
+size 4983398004

pytorch_model-00002-of-00002.bin → pytorch_model-00003-of-00003.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:14c74cc690ee7006a16c32655d698367209d2739396c548ca02b6fc8aef2eaef
+oid sha256:653556b3b21b03d7198cb94d6a4405d2e3cb6b329a06cc4333c710e9f43a1e45
 size 4993663292

special_tokens_map.json

@@ -1,5 +1,23 @@
 {
-  "eos_token": "</s>",
-  "pad_token": "<pad>",
-  "unk_token": "<unk>"
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
 }

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6502d07619068a98aa2d3bb531332a694ffe108ca6c6fe62a467ccfe98d666b9
-size 16315219
+oid sha256:54e5c72a5ea09da48b2f316760b8bc5a445683ab9a5bc6b68db5d8db624ecceb
+size 16315213

tokenizer_config.json

@@ -1,5 +1,31 @@
 {
-  "additional_special_tokens": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
   "clean_up_tokenization_spaces": true,
   "eos_token": "</s>",
   "extra_ids": 0,

train_results.json

@@ -1,8 +1,8 @@
 {
-    "epoch": 8.0,
-    "train_loss": 1.429130665848895,
-    "train_runtime": 9768.8261,
+    "epoch": 15.0,
+    "train_loss": 1.153788715837463,
+    "train_runtime": 20083.7121,
     "train_samples": 65584,
-    "train_samples_per_second": 134.272,
-    "train_steps_per_second": 1.048
+    "train_samples_per_second": 65.311,
+    "train_steps_per_second": 0.51
 }

trainer_state.json

@@ -1,8 +1,9 @@
 {
-  "best_metric": 0.2082,
-  "best_model_checkpoint": "mt0-xl_russian_natprompt_adafactor/checkpoint-2562",
-  "epoch": 8.0,
-  "global_step": 4100,
+  "best_metric": 17.3273,
+  "best_model_checkpoint": "models/mt0-xl_russian_natprompt_adafactor_updated/checkpoint-6150",
+  "epoch": 14.999024390243903,
+  "eval_steps": 500,
+  "global_step": 7687,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
@@ -10,168 +11,305 @@
     {
       "epoch": 1.0,
       "learning_rate": 4.75e-05,
-      "loss": 2.0449,
+      "loss": 2.0388,
       "step": 512
     },
     {
       "epoch": 1.0,
-      "eval_gen_len": 16.75809549945115,
-      "eval_loss": 1.6754746437072754,
-      "eval_rouge1": 0.0817,
-      "eval_rouge2": 0.0,
-      "eval_rougeL": 0.0778,
-      "eval_rougeLsum": 0.0817,
-      "eval_runtime": 196.0938,
-      "eval_samples_per_second": 37.166,
-      "eval_steps_per_second": 1.163,
+      "eval_gen_len": 16.58484100877193,
+      "eval_loss": 1.6734575033187866,
+      "eval_rouge1": 14.1367,
+      "eval_rouge2": 7.0437,
+      "eval_rougeL": 14.0625,
+      "eval_rougeLsum": 14.0916,
+      "eval_runtime": 270.6111,
+      "eval_samples_per_second": 26.932,
+      "eval_steps_per_second": 0.843,
       "step": 512
     },
     {
       "epoch": 2.0,
       "learning_rate": 4.4995117187500005e-05,
-      "loss": 1.707,
+      "loss": 1.7098,
       "step": 1025
     },
     {
       "epoch": 2.0,
-      "eval_gen_len": 15.862102085620197,
-      "eval_loss": 1.6181610822677612,
-      "eval_rouge1": 0.096,
-      "eval_rouge2": 0.0,
-      "eval_rougeL": 0.097,
-      "eval_rougeLsum": 0.1,
-      "eval_runtime": 171.5849,
-      "eval_samples_per_second": 42.475,
-      "eval_steps_per_second": 1.329,
+      "eval_gen_len": 16.68050986842105,
+      "eval_loss": 1.6203718185424805,
+      "eval_rouge1": 15.2619,
+      "eval_rouge2": 7.8124,
+      "eval_rougeL": 15.159,
+      "eval_rougeLsum": 15.2078,
+      "eval_runtime": 276.6842,
+      "eval_samples_per_second": 26.341,
+      "eval_steps_per_second": 0.824,
       "step": 1025
     },
     {
       "epoch": 3.0,
       "learning_rate": 4.24951171875e-05,
-      "loss": 1.5398,
+      "loss": 1.539,
       "step": 1537
     },
     {
       "epoch": 3.0,
-      "eval_gen_len": 16.47653677277717,
-      "eval_loss": 1.6085278987884521,
-      "eval_rouge1": 0.1394,
-      "eval_rouge2": 0.0034,
-      "eval_rougeL": 0.1401,
-      "eval_rougeLsum": 0.1416,
-      "eval_runtime": 171.9932,
-      "eval_samples_per_second": 42.374,
-      "eval_steps_per_second": 1.326,
+      "eval_gen_len": 16.61417214912281,
+      "eval_loss": 1.6058766841888428,
+      "eval_rouge1": 15.9942,
+      "eval_rouge2": 8.1827,
+      "eval_rougeL": 15.872,
+      "eval_rougeLsum": 15.9105,
+      "eval_runtime": 263.8074,
+      "eval_samples_per_second": 27.626,
+      "eval_steps_per_second": 0.864,
       "step": 1537
     },
     {
       "epoch": 4.0,
       "learning_rate": 3.9990234375e-05,
-      "loss": 1.4142,
+      "loss": 1.403,
       "step": 2050
     },
     {
       "epoch": 4.0,
-      "eval_gen_len": 16.273188803512625,
-      "eval_loss": 1.6016370058059692,
-      "eval_rouge1": 0.1132,
-      "eval_rouge2": 0.0,
-      "eval_rougeL": 0.1132,
-      "eval_rougeLsum": 0.1098,
-      "eval_runtime": 171.2054,
-      "eval_samples_per_second": 42.569,
-      "eval_steps_per_second": 1.332,
+      "eval_gen_len": 16.26343201754386,
+      "eval_loss": 1.6041721105575562,
+      "eval_rouge1": 16.6383,
+      "eval_rouge2": 8.4603,
+      "eval_rougeL": 16.5096,
+      "eval_rougeLsum": 16.5635,
+      "eval_runtime": 251.4581,
+      "eval_samples_per_second": 28.983,
+      "eval_steps_per_second": 0.907,
       "step": 2050
     },
     {
       "epoch": 5.0,
       "learning_rate": 3.7490234375e-05,
-      "loss": 1.3102,
+      "loss": 1.295,
       "step": 2562
     },
     {
       "epoch": 5.0,
-      "eval_gen_len": 16.287733260153676,
-      "eval_loss": 1.6240657567977905,
-      "eval_rouge1": 0.2082,
-      "eval_rouge2": 0.0034,
-      "eval_rougeL": 0.2054,
-      "eval_rougeLsum": 0.2061,
-      "eval_runtime": 170.3025,
-      "eval_samples_per_second": 42.794,
-      "eval_steps_per_second": 1.339,
+      "eval_gen_len": 15.741365131578947,
+      "eval_loss": 1.6226089000701904,
+      "eval_rouge1": 16.9189,
+      "eval_rouge2": 8.8384,
+      "eval_rougeL": 16.7799,
+      "eval_rougeLsum": 16.8258,
+      "eval_runtime": 169.6881,
+      "eval_samples_per_second": 42.949,
+      "eval_steps_per_second": 1.344,
       "step": 2562
     },
     {
       "epoch": 6.0,
       "learning_rate": 3.49853515625e-05,
-      "loss": 1.2162,
+      "loss": 1.1984,
       "step": 3075
     },
     {
       "epoch": 6.0,
-      "eval_gen_len": 16.158068057080133,
-      "eval_loss": 1.6281158924102783,
-      "eval_rouge1": 0.1549,
-      "eval_rouge2": 0.0,
-      "eval_rougeL": 0.1549,
-      "eval_rougeLsum": 0.1549,
-      "eval_runtime": 171.3659,
-      "eval_samples_per_second": 42.529,
-      "eval_steps_per_second": 1.33,
+      "eval_gen_len": 15.888157894736842,
+      "eval_loss": 1.6289030313491821,
+      "eval_rouge1": 16.9788,
+      "eval_rouge2": 8.7272,
+      "eval_rougeL": 16.8238,
+      "eval_rougeLsum": 16.8765,
+      "eval_runtime": 175.0677,
+      "eval_samples_per_second": 41.63,
+      "eval_steps_per_second": 1.302,
       "step": 3075
     },
     {
       "epoch": 7.0,
       "learning_rate": 3.2485351562499996e-05,
-      "loss": 1.1364,
+      "loss": 1.1195,
       "step": 3587
     },
     {
       "epoch": 7.0,
-      "eval_gen_len": 15.992453347969265,
-      "eval_loss": 1.6622037887573242,
-      "eval_rouge1": 0.1583,
-      "eval_rouge2": 0.0,
-      "eval_rougeL": 0.1575,
-      "eval_rougeLsum": 0.1589,
-      "eval_runtime": 254.3332,
-      "eval_samples_per_second": 28.655,
-      "eval_steps_per_second": 0.896,
+      "eval_gen_len": 16.23519736842105,
+      "eval_loss": 1.6697918176651,
+      "eval_rouge1": 17.0912,
+      "eval_rouge2": 8.7061,
+      "eval_rougeL": 16.9084,
+      "eval_rougeLsum": 16.9633,
+      "eval_runtime": 171.9395,
+      "eval_samples_per_second": 42.387,
+      "eval_steps_per_second": 1.326,
       "step": 3587
     },
     {
       "epoch": 8.0,
       "learning_rate": 2.998046875e-05,
-      "loss": 1.0649,
+      "loss": 1.0463,
       "step": 4100
     },
     {
       "epoch": 8.0,
-      "eval_gen_len": 16.509879253567508,
-      "eval_loss": 1.6811630725860596,
-      "eval_rouge1": 0.2033,
-      "eval_rouge2": 0.0137,
-      "eval_rougeL": 0.2012,
-      "eval_rougeLsum": 0.2027,
-      "eval_runtime": 173.1353,
-      "eval_samples_per_second": 42.094,
-      "eval_steps_per_second": 1.317,
+      "eval_gen_len": 16.14761513157895,
+      "eval_loss": 1.6845269203186035,
+      "eval_rouge1": 17.201,
+      "eval_rouge2": 8.7395,
+      "eval_rougeL": 17.003,
+      "eval_rougeLsum": 17.052,
+      "eval_runtime": 252.7052,
+      "eval_samples_per_second": 28.84,
+      "eval_steps_per_second": 0.902,
       "step": 4100
     },
     {
-      "epoch": 8.0,
-      "step": 4100,
-      "total_flos": 9.102827646479237e+17,
-      "train_loss": 1.429130665848895,
-      "train_runtime": 9768.8261,
-      "train_samples_per_second": 134.272,
-      "train_steps_per_second": 1.048
+      "epoch": 9.0,
+      "learning_rate": 2.748046875e-05,
+      "loss": 0.9866,
+      "step": 4612
+    },
+    {
+      "epoch": 9.0,
+      "eval_gen_len": 15.878837719298245,
+      "eval_loss": 1.726230502128601,
+      "eval_rouge1": 17.3223,
+      "eval_rouge2": 8.8289,
+      "eval_rougeL": 17.1413,
+      "eval_rougeLsum": 17.1756,
+      "eval_runtime": 182.5703,
+      "eval_samples_per_second": 39.919,
+      "eval_steps_per_second": 1.249,
+      "step": 4612
+    },
+    {
+      "epoch": 10.0,
+      "learning_rate": 2.49755859375e-05,
+      "loss": 0.9326,
+      "step": 5125
+    },
+    {
+      "epoch": 10.0,
+      "eval_gen_len": 15.797149122807017,
+      "eval_loss": 1.7532711029052734,
+      "eval_rouge1": 17.2655,
+      "eval_rouge2": 8.7512,
+      "eval_rougeL": 17.0508,
+      "eval_rougeLsum": 17.1055,
+      "eval_runtime": 168.7949,
+      "eval_samples_per_second": 43.177,
+      "eval_steps_per_second": 1.351,
+      "step": 5125
+    },
+    {
+      "epoch": 11.0,
+      "learning_rate": 2.24755859375e-05,
+      "loss": 0.8844,
+      "step": 5637
+    },
+    {
+      "epoch": 11.0,
+      "eval_gen_len": 16.32360197368421,
+      "eval_loss": 1.7794246673583984,
+      "eval_rouge1": 17.008,
+      "eval_rouge2": 8.5404,
+      "eval_rougeL": 16.8044,
+      "eval_rougeLsum": 16.848,
+      "eval_runtime": 168.6102,
+      "eval_samples_per_second": 43.224,
+      "eval_steps_per_second": 1.352,
+      "step": 5637
+    },
+    {
+      "epoch": 12.0,
+      "learning_rate": 1.9970703125e-05,
+      "loss": 0.8393,
+      "step": 6150
+    },
+    {
+      "epoch": 12.0,
+      "eval_gen_len": 16.143092105263158,
+      "eval_loss": 1.7995822429656982,
+      "eval_rouge1": 17.3273,
+      "eval_rouge2": 8.7829,
+      "eval_rougeL": 17.097,
+      "eval_rougeLsum": 17.1644,
+      "eval_runtime": 171.5723,
+      "eval_samples_per_second": 42.478,
+      "eval_steps_per_second": 1.329,
+      "step": 6150
+    },
+    {
+      "epoch": 13.0,
+      "learning_rate": 1.7470703125000003e-05,
+      "loss": 0.8046,
+      "step": 6662
+    },
+    {
+      "epoch": 13.0,
+      "eval_gen_len": 16.090597587719298,
+      "eval_loss": 1.8266295194625854,
+      "eval_rouge1": 17.1859,
+      "eval_rouge2": 8.6524,
+      "eval_rougeL": 16.9605,
+      "eval_rougeLsum": 17.0118,
+      "eval_runtime": 259.1646,
+      "eval_samples_per_second": 28.121,
+      "eval_steps_per_second": 0.88,
+      "step": 6662
+    },
+    {
+      "epoch": 14.0,
+      "learning_rate": 1.49658203125e-05,
+      "loss": 0.7682,
+      "step": 7175
+    },
+    {
+      "epoch": 14.0,
+      "eval_gen_len": 16.11239035087719,
+      "eval_loss": 1.8624775409698486,
+      "eval_rouge1": 17.0184,
+      "eval_rouge2": 8.5314,
+      "eval_rougeL": 16.8019,
+      "eval_rougeLsum": 16.847,
+      "eval_runtime": 170.9938,
+      "eval_samples_per_second": 42.621,
+      "eval_steps_per_second": 1.333,
+      "step": 7175
+    },
+    {
+      "epoch": 15.0,
+      "learning_rate": 1.2465820312500002e-05,
+      "loss": 0.7419,
+      "step": 7687
+    },
+    {
+      "epoch": 15.0,
+      "eval_gen_len": 15.95751096491228,
+      "eval_loss": 1.8779526948928833,
+      "eval_rouge1": 17.2742,
+      "eval_rouge2": 8.6795,
+      "eval_rougeL": 17.0699,
+      "eval_rougeLsum": 17.1118,
+      "eval_runtime": 177.9916,
+      "eval_samples_per_second": 40.946,
+      "eval_steps_per_second": 1.281,
+      "step": 7687
+    },
+    {
+      "epoch": 15.0,
+      "step": 7687,
+      "total_flos": 1.7085595424946913e+18,
+      "train_loss": 1.153788715837463,
+      "train_runtime": 20083.7121,
+      "train_samples_per_second": 65.311,
+      "train_steps_per_second": 0.51
     }
   ],
+  "logging_steps": 500,
   "max_steps": 10240,
+  "num_input_tokens_seen": 0,
   "num_train_epochs": 20,
-  "total_flos": 9.102827646479237e+17,
+  "save_steps": 500,
+  "total_flos": 1.7085595424946913e+18,
+  "train_batch_size": 4,
   "trial_name": null,
   "trial_params": null
 }

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cd882549d0b55e5e64fc1bccbe7a30e73de399085741636b5d21642c59b40dda
-size 4091
+oid sha256:b05c536c471b5be16fe45bc4130c51587d993402ce5ba1bd7ba28b30f5b50b5b
+size 4411

upload.py

@@ -1,11 +0,0 @@
-#!/bin/env python3
-
-import sys
-from huggingface_hub import HfApi
-from huggingface_hub import create_repo
-
-create_repo(sys.argv[1])
-api = HfApi()
-
-api.upload_folder(folder_path=".", repo_id=sys.argv[1], repo_type="model")
-