File size: 4,489 Bytes
e2a13bd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
---
license: apache-2.0
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
datasets:
- generator
base_model: mistralai/Mistral-7B-Instruct-v0.2
model-index:
- name: mistralai/Mistral-7B-Instruct-v0.2
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# mistralai/Mistral-7B-Instruct-v0.2
This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) on the generator dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5526
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2.5e-05
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 0.03
- training_steps: 600
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.7925 | 0.22 | 10 | 2.0998 |
| 1.6897 | 0.43 | 20 | 1.3864 |
| 1.3495 | 0.65 | 30 | 1.2622 |
| 1.2144 | 0.87 | 40 | 1.1882 |
| 1.1546 | 1.09 | 50 | 1.1397 |
| 1.1002 | 1.3 | 60 | 1.0843 |
| 1.0023 | 1.52 | 70 | 0.9794 |
| 0.897 | 1.74 | 80 | 0.9370 |
| 0.8625 | 1.96 | 90 | 0.8557 |
| 0.7492 | 2.17 | 100 | 0.7909 |
| 0.7296 | 2.39 | 110 | 0.7455 |
| 0.6738 | 2.61 | 120 | 0.7239 |
| 0.656 | 2.83 | 130 | 0.7071 |
| 0.6289 | 3.04 | 140 | 0.6852 |
| 0.5835 | 3.26 | 150 | 0.6704 |
| 0.5647 | 3.48 | 160 | 0.6481 |
| 0.5416 | 3.7 | 170 | 0.6326 |
| 0.5159 | 3.91 | 180 | 0.6219 |
| 0.475 | 4.13 | 190 | 0.6091 |
| 0.4529 | 4.35 | 200 | 0.5903 |
| 0.4358 | 4.57 | 210 | 0.5769 |
| 0.4124 | 4.78 | 220 | 0.5574 |
| 0.3925 | 5.0 | 230 | 0.5433 |
| 0.3325 | 5.22 | 240 | 0.5396 |
| 0.3307 | 5.43 | 250 | 0.5241 |
| 0.3122 | 5.65 | 260 | 0.5185 |
| 0.2973 | 5.87 | 270 | 0.5042 |
| 0.2695 | 6.09 | 280 | 0.5082 |
| 0.2345 | 6.3 | 290 | 0.5020 |
| 0.2307 | 6.52 | 300 | 0.4859 |
| 0.2226 | 6.74 | 310 | 0.4771 |
| 0.2083 | 6.96 | 320 | 0.4717 |
| 0.1858 | 7.17 | 330 | 0.4881 |
| 0.1677 | 7.39 | 340 | 0.4791 |
| 0.1663 | 7.61 | 350 | 0.4774 |
| 0.1609 | 7.83 | 360 | 0.4780 |
| 0.1493 | 8.04 | 370 | 0.4820 |
| 0.1332 | 8.26 | 380 | 0.4940 |
| 0.1351 | 8.48 | 390 | 0.4898 |
| 0.1251 | 8.7 | 400 | 0.4894 |
| 0.1243 | 8.91 | 410 | 0.4836 |
| 0.1121 | 9.13 | 420 | 0.5108 |
| 0.1059 | 9.35 | 430 | 0.5055 |
| 0.1037 | 9.57 | 440 | 0.4974 |
| 0.102 | 9.78 | 450 | 0.4981 |
| 0.1032 | 10.0 | 460 | 0.5100 |
| 0.0887 | 10.22 | 470 | 0.5267 |
| 0.09 | 10.43 | 480 | 0.5231 |
| 0.084 | 10.65 | 490 | 0.5228 |
| 0.0865 | 10.87 | 500 | 0.5166 |
| 0.0838 | 11.09 | 510 | 0.5337 |
| 0.0762 | 11.3 | 520 | 0.5444 |
| 0.0792 | 11.52 | 530 | 0.5375 |
| 0.0765 | 11.74 | 540 | 0.5397 |
| 0.0747 | 11.96 | 550 | 0.5386 |
| 0.0684 | 12.17 | 560 | 0.5517 |
| 0.0697 | 12.39 | 570 | 0.5547 |
| 0.0701 | 12.61 | 580 | 0.5528 |
| 0.0702 | 12.83 | 590 | 0.5522 |
| 0.0693 | 13.04 | 600 | 0.5526 |
### Framework versions
- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.16.1
- Tokenizers 0.15.0 |