Training in progress, step 50, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: tiiuae/falcon-rw-1b
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "tiiuae/falcon-rw-1b",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "dense_h_to_4h",
+    "query_key_value",
+    "dense",
+    "dense_4h_to_h"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8d29530112216e70e3f7241a5b2e265347bb259ba2e67ee0ccd40a6f674c1b6
+size 50358592

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbd84331b83484116e005eaacd7816a2111ffe18f4becd691c3658a23a2d2122
+size 100824826

last-checkpoint/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:026b434bd849edd3081dacbb7e6982b76fb8bbdda0a5b6558879ae65604616b0
+size 14244

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e55e0dc9a04f69798703b4b7992f93d80a36094516db17aac03e75df3075b1a
+size 1064

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,416 @@
+{
+  "best_metric": 1.018844723701477,
+  "best_model_checkpoint": "miner_id_24/checkpoint-50",
+  "epoch": 0.5813953488372093,
+  "eval_steps": 25,
+  "global_step": 50,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.011627906976744186,
+      "grad_norm": 2.9615676403045654,
+      "learning_rate": 2.9999999999999997e-05,
+      "loss": 3.7306,
+      "step": 1
+    },
+    {
+      "epoch": 0.011627906976744186,
+      "eval_loss": 1.230737328529358,
+      "eval_runtime": 7.9791,
+      "eval_samples_per_second": 6.266,
+      "eval_steps_per_second": 0.877,
+      "step": 1
+    },
+    {
+      "epoch": 0.023255813953488372,
+      "grad_norm": 2.557929754257202,
+      "learning_rate": 5.9999999999999995e-05,
+      "loss": 3.564,
+      "step": 2
+    },
+    {
+      "epoch": 0.03488372093023256,
+      "grad_norm": 2.1619441509246826,
+      "learning_rate": 8.999999999999999e-05,
+      "loss": 3.744,
+      "step": 3
+    },
+    {
+      "epoch": 0.046511627906976744,
+      "grad_norm": 1.1338863372802734,
+      "learning_rate": 0.00011999999999999999,
+      "loss": 5.1335,
+      "step": 4
+    },
+    {
+      "epoch": 0.05813953488372093,
+      "grad_norm": 1.0265060663223267,
+      "learning_rate": 0.00015,
+      "loss": 4.7372,
+      "step": 5
+    },
+    {
+      "epoch": 0.06976744186046512,
+      "grad_norm": 1.4262174367904663,
+      "learning_rate": 0.00017999999999999998,
+      "loss": 5.1326,
+      "step": 6
+    },
+    {
+      "epoch": 0.08139534883720931,
+      "grad_norm": 1.4468605518341064,
+      "learning_rate": 0.00020999999999999998,
+      "loss": 5.1196,
+      "step": 7
+    },
+    {
+      "epoch": 0.09302325581395349,
+      "grad_norm": 1.4061644077301025,
+      "learning_rate": 0.00023999999999999998,
+      "loss": 5.5418,
+      "step": 8
+    },
+    {
+      "epoch": 0.10465116279069768,
+      "grad_norm": 1.197460651397705,
+      "learning_rate": 0.00027,
+      "loss": 4.9912,
+      "step": 9
+    },
+    {
+      "epoch": 0.11627906976744186,
+      "grad_norm": 1.332681655883789,
+      "learning_rate": 0.0003,
+      "loss": 5.1568,
+      "step": 10
+    },
+    {
+      "epoch": 0.12790697674418605,
+      "grad_norm": 1.0973302125930786,
+      "learning_rate": 0.00029997179556727515,
+      "loss": 5.2029,
+      "step": 11
+    },
+    {
+      "epoch": 0.13953488372093023,
+      "grad_norm": 1.9083119630813599,
+      "learning_rate": 0.0002998871928756345,
+      "loss": 4.6768,
+      "step": 12
+    },
+    {
+      "epoch": 0.1511627906976744,
+      "grad_norm": 1.5310242176055908,
+      "learning_rate": 0.00029974622374069024,
+      "loss": 4.4159,
+      "step": 13
+    },
+    {
+      "epoch": 0.16279069767441862,
+      "grad_norm": 0.9514783620834351,
+      "learning_rate": 0.0002995489411751688,
+      "loss": 4.3375,
+      "step": 14
+    },
+    {
+      "epoch": 0.1744186046511628,
+      "grad_norm": 1.3517793416976929,
+      "learning_rate": 0.0002992954193689748,
+      "loss": 4.7541,
+      "step": 15
+    },
+    {
+      "epoch": 0.18604651162790697,
+      "grad_norm": 1.600083827972412,
+      "learning_rate": 0.00029898575366129145,
+      "loss": 5.5065,
+      "step": 16
+    },
+    {
+      "epoch": 0.19767441860465115,
+      "grad_norm": 0.972209095954895,
+      "learning_rate": 0.00029862006050472675,
+      "loss": 4.8114,
+      "step": 17
+    },
+    {
+      "epoch": 0.20930232558139536,
+      "grad_norm": 0.8742607831954956,
+      "learning_rate": 0.0002981984774215213,
+      "loss": 4.5777,
+      "step": 18
+    },
+    {
+      "epoch": 0.22093023255813954,
+      "grad_norm": 1.0211178064346313,
+      "learning_rate": 0.0002977211629518312,
+      "loss": 5.0349,
+      "step": 19
+    },
+    {
+      "epoch": 0.23255813953488372,
+      "grad_norm": 1.19241464138031,
+      "learning_rate": 0.00029718829659410766,
+      "loss": 5.1001,
+      "step": 20
+    },
+    {
+      "epoch": 0.2441860465116279,
+      "grad_norm": 2.0463316440582275,
+      "learning_rate": 0.00029660007873759533,
+      "loss": 5.5808,
+      "step": 21
+    },
+    {
+      "epoch": 0.2558139534883721,
+      "grad_norm": 5.027365207672119,
+      "learning_rate": 0.00029595673058697357,
+      "loss": 3.2185,
+      "step": 22
+    },
+    {
+      "epoch": 0.26744186046511625,
+      "grad_norm": 1.7920422554016113,
+      "learning_rate": 0.00029525849407917087,
+      "loss": 3.5772,
+      "step": 23
+    },
+    {
+      "epoch": 0.27906976744186046,
+      "grad_norm": 2.2887630462646484,
+      "learning_rate": 0.000294505631792382,
+      "loss": 2.6486,
+      "step": 24
+    },
+    {
+      "epoch": 0.29069767441860467,
+      "grad_norm": 1.6574441194534302,
+      "learning_rate": 0.00029369842684732334,
+      "loss": 3.5684,
+      "step": 25
+    },
+    {
+      "epoch": 0.29069767441860467,
+      "eval_loss": 1.0823593139648438,
+      "eval_runtime": 7.799,
+      "eval_samples_per_second": 6.411,
+      "eval_steps_per_second": 0.898,
+      "step": 25
+    },
+    {
+      "epoch": 0.3023255813953488,
+      "grad_norm": 0.8784985542297363,
+      "learning_rate": 0.00029283718280076227,
+      "loss": 4.3739,
+      "step": 26
+    },
+    {
+      "epoch": 0.313953488372093,
+      "grad_norm": 1.098872423171997,
+      "learning_rate": 0.00029192222353136254,
+      "loss": 4.6181,
+      "step": 27
+    },
+    {
+      "epoch": 0.32558139534883723,
+      "grad_norm": 0.9574615359306335,
+      "learning_rate": 0.0002909538931178862,
+      "loss": 4.7032,
+      "step": 28
+    },
+    {
+      "epoch": 0.3372093023255814,
+      "grad_norm": 1.1205692291259766,
+      "learning_rate": 0.0002899325557098001,
+      "loss": 4.9195,
+      "step": 29
+    },
+    {
+      "epoch": 0.3488372093023256,
+      "grad_norm": 0.9449394941329956,
+      "learning_rate": 0.00028885859539033357,
+      "loss": 4.2665,
+      "step": 30
+    },
+    {
+      "epoch": 0.36046511627906974,
+      "grad_norm": 1.0518043041229248,
+      "learning_rate": 0.0002877324160320411,
+      "loss": 4.6985,
+      "step": 31
+    },
+    {
+      "epoch": 0.37209302325581395,
+      "grad_norm": 0.9868097901344299,
+      "learning_rate": 0.000286554441144922,
+      "loss": 4.8123,
+      "step": 32
+    },
+    {
+      "epoch": 0.38372093023255816,
+      "grad_norm": 0.8500082492828369,
+      "learning_rate": 0.00028532511371715566,
+      "loss": 4.5633,
+      "step": 33
+    },
+    {
+      "epoch": 0.3953488372093023,
+      "grad_norm": 0.9240705370903015,
+      "learning_rate": 0.0002840448960485118,
+      "loss": 4.4635,
+      "step": 34
+    },
+    {
+      "epoch": 0.4069767441860465,
+      "grad_norm": 0.8523270487785339,
+      "learning_rate": 0.00028271426957649865,
+      "loss": 4.1489,
+      "step": 35
+    },
+    {
+      "epoch": 0.4186046511627907,
+      "grad_norm": 0.9723194241523743,
+      "learning_rate": 0.00028133373469531363,
+      "loss": 4.0247,
+      "step": 36
+    },
+    {
+      "epoch": 0.43023255813953487,
+      "grad_norm": 1.9848984479904175,
+      "learning_rate": 0.0002799038105676658,
+      "loss": 4.588,
+      "step": 37
+    },
+    {
+      "epoch": 0.4418604651162791,
+      "grad_norm": 0.8572053909301758,
+      "learning_rate": 0.00027842503492953995,
+      "loss": 4.6031,
+      "step": 38
+    },
+    {
+      "epoch": 0.45348837209302323,
+      "grad_norm": 0.9035416841506958,
+      "learning_rate": 0.0002768979638879761,
+      "loss": 4.8538,
+      "step": 39
+    },
+    {
+      "epoch": 0.46511627906976744,
+      "grad_norm": 1.053252935409546,
+      "learning_rate": 0.00027532317171194046,
+      "loss": 4.9722,
+      "step": 40
+    },
+    {
+      "epoch": 0.47674418604651164,
+      "grad_norm": 1.071892261505127,
+      "learning_rate": 0.000273701250616366,
+      "loss": 5.0359,
+      "step": 41
+    },
+    {
+      "epoch": 0.4883720930232558,
+      "grad_norm": 2.722141981124878,
+      "learning_rate": 0.0002720328105394451,
+      "loss": 6.2443,
+      "step": 42
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 2.1856935024261475,
+      "learning_rate": 0.00027031847891325657,
+      "loss": 3.1716,
+      "step": 43
+    },
+    {
+      "epoch": 0.5116279069767442,
+      "grad_norm": 1.6351940631866455,
+      "learning_rate": 0.00026855890042781387,
+      "loss": 2.7951,
+      "step": 44
+    },
+    {
+      "epoch": 0.5232558139534884,
+      "grad_norm": 1.7642279863357544,
+      "learning_rate": 0.000266754736788624,
+      "loss": 3.6811,
+      "step": 45
+    },
+    {
+      "epoch": 0.5348837209302325,
+      "grad_norm": 1.665910243988037,
+      "learning_rate": 0.00026490666646784665,
+      "loss": 4.2234,
+      "step": 46
+    },
+    {
+      "epoch": 0.5465116279069767,
+      "grad_norm": 1.4838505983352661,
+      "learning_rate": 0.00026301538444914907,
+      "loss": 4.4947,
+      "step": 47
+    },
+    {
+      "epoch": 0.5581395348837209,
+      "grad_norm": 1.082304835319519,
+      "learning_rate": 0.00026108160196635066,
+      "loss": 4.6096,
+      "step": 48
+    },
+    {
+      "epoch": 0.5697674418604651,
+      "grad_norm": 0.9804915189743042,
+      "learning_rate": 0.0002591060462359573,
+      "loss": 4.3917,
+      "step": 49
+    },
+    {
+      "epoch": 0.5813953488372093,
+      "grad_norm": 1.103212594985962,
+      "learning_rate": 0.00025708946018368484,
+      "loss": 4.2729,
+      "step": 50
+    },
+    {
+      "epoch": 0.5813953488372093,
+      "eval_loss": 1.018844723701477,
+      "eval_runtime": 8.0091,
+      "eval_samples_per_second": 6.243,
+      "eval_steps_per_second": 0.874,
+      "step": 50
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 172,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 1,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.2576418474950656e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c9313898ff0abfec328e06288a72fcd0230945d5f1c5a25da7eff28ed43dcf5
+size 6840