Hugging Face's logo

parth0908
/
peft-starcoder-finetuned

PEFT
TensorBoard
Safetensors
gpt_bigcode
Generated from Trainer
Model card Files Metrics Community
parth0908 commited on
Commit
5d22897
•
1 Parent(s): 0ae049b

Training in progress, step 500

Browse files
Files changed (16) hide show
  1. README.md +37 -17
  2. adapter_config.json +2 -2
  3. adapter_model.safetensors +1 -1
  4. generation_config.json +6 -0
  5. runs/Nov19_13-49-20_localhost/events.out.tfevents.1732004362.localhost +3 -0
  6. runs/Nov19_13-52-01_localhost/events.out.tfevents.1732004522.localhost +3 -0
  7. runs/Nov19_14-03-23_localhost/events.out.tfevents.1732005204.localhost +3 -0
  8. runs/Nov19_14-13-30_localhost/events.out.tfevents.1732005811.localhost +3 -0
  9. runs/Nov21_13-53-49_localhost/events.out.tfevents.1732177431.localhost +3 -0
  10. runs/Nov21_16-15-04_localhost/events.out.tfevents.1732185906.localhost +3 -0
  11. runs/Nov21_17-19-19_localhost/events.out.tfevents.1732189761.localhost +3 -0
  12. runs/Nov21_17-23-58_localhost/events.out.tfevents.1732190040.localhost +3 -0
  13. runs/Nov21_18-47-25_localhost/events.out.tfevents.1732195047.localhost +3 -0
  14. runs/Nov21_19-13-55_localhost/events.out.tfevents.1732196637.localhost +3 -0
  15. runs/Nov21_20-47-21_localhost/events.out.tfevents.1732202243.localhost +3 -0
  16. training_args.bin +1 -1
README.md CHANGED
@@ -1,7 +1,7 @@
1
  ---
2
- base_model: bigcode/starcoderbase-1b
3
- library_name: peft
4
  license: bigcode-openrail-m
 
5
  tags:
6
  - generated_from_trainer
7
  model-index:
@@ -14,14 +14,9 @@ should probably proofread and complete it, then remove this comment. -->
14
 
15
  # peft-starcoder-finetuned
16
 
17
- This model is a fine-tuned version of [bigcode/starcoderbase-1b](https://huggingface.co/bigcode/starcoderbase-1b) on an unknown dataset.
18
  It achieves the following results on the evaluation set:
19
- - eval_loss: nan
20
- - eval_runtime: 51.0638
21
- - eval_samples_per_second: 1.43
22
- - eval_steps_per_second: 1.43
23
- - epoch: 0.3
24
- - step: 60
25
 
26
  ## Model description
27
 
@@ -40,21 +35,46 @@ More information needed
40
  ### Training hyperparameters
41
 
42
  The following hyperparameters were used during training:
43
- - learning_rate: 0.0005
44
- - train_batch_size: 1
45
- - eval_batch_size: 1
46
  - seed: 42
47
  - gradient_accumulation_steps: 8
48
- - total_train_batch_size: 8
49
  - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
50
  - lr_scheduler_type: cosine
51
- - lr_scheduler_warmup_steps: 10
52
- - training_steps: 200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
 
54
  ### Framework versions
55
 
56
- - PEFT 0.13.2
57
  - Transformers 4.46.2
58
  - Pytorch 2.4.1+cu121
59
  - Datasets 3.0.1
60
- - Tokenizers 0.20.3
 
1
  ---
2
+ library_name: transformers
 
3
  license: bigcode-openrail-m
4
+ base_model: bigcode/starcoderbase-1b
5
  tags:
6
  - generated_from_trainer
7
  model-index:
 
14
 
15
  # peft-starcoder-finetuned
16
 
17
+ This model is a fine-tuned version of [bigcode/starcoderbase-1b](https://huggingface.co/bigcode/starcoderbase-1b) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
+ - Loss: 1.5923
 
 
 
 
 
20
 
21
  ## Model description
22
 
 
35
  ### Training hyperparameters
36
 
37
  The following hyperparameters were used during training:
38
+ - learning_rate: 5e-05
39
+ - train_batch_size: 2
40
+ - eval_batch_size: 2
41
  - seed: 42
42
  - gradient_accumulation_steps: 8
43
+ - total_train_batch_size: 16
44
  - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
45
  - lr_scheduler_type: cosine
46
+ - lr_scheduler_warmup_steps: 50
47
+ - training_steps: 2000
48
+
49
+ ### Training results
50
+
51
+ | Training Loss | Epoch | Step | Validation Loss |
52
+ |:-------------:|:------:|:----:|:---------------:|
53
+ | 0.5622 | 0.1992 | 100 | 1.2485 |
54
+ | 0.4589 | 0.3984 | 200 | 1.2126 |
55
+ | 0.4216 | 0.5976 | 300 | 1.2730 |
56
+ | 0.3743 | 0.7968 | 400 | 1.2278 |
57
+ | 0.3535 | 0.9960 | 500 | 1.2615 |
58
+ | 0.3011 | 1.1952 | 600 | 1.2960 |
59
+ | 0.2653 | 1.3944 | 700 | 1.3112 |
60
+ | 0.2734 | 1.5936 | 800 | 1.3759 |
61
+ | 0.2855 | 1.7928 | 900 | 1.3015 |
62
+ | 0.2528 | 1.9920 | 1000 | 1.3470 |
63
+ | 0.2083 | 2.1912 | 1100 | 1.4719 |
64
+ | 0.2318 | 2.3904 | 1200 | 1.4494 |
65
+ | 0.1935 | 2.5896 | 1300 | 1.4621 |
66
+ | 0.1809 | 2.7888 | 1400 | 1.4829 |
67
+ | 0.227 | 2.9880 | 1500 | 1.4911 |
68
+ | 0.1813 | 3.1873 | 1600 | 1.5903 |
69
+ | 0.1893 | 3.3865 | 1700 | 1.5906 |
70
+ | 0.1674 | 3.5857 | 1800 | 1.5916 |
71
+ | 0.1723 | 3.7849 | 1900 | 1.5921 |
72
+ | 0.1843 | 3.9841 | 2000 | 1.5923 |
73
+
74
 
75
  ### Framework versions
76
 
 
77
  - Transformers 4.46.2
78
  - Pytorch 2.4.1+cu121
79
  - Datasets 3.0.1
80
+ - Tokenizers 0.20.3
adapter_config.json CHANGED
@@ -21,9 +21,9 @@
21
  "revision": null,
22
  "target_modules": [
23
  "c_proj",
24
- "c_attn",
25
  "q_attn",
26
- "c_fc"
 
27
  ],
28
  "task_type": "CAUSAL_LM",
29
  "use_dora": false,
 
21
  "revision": null,
22
  "target_modules": [
23
  "c_proj",
 
24
  "q_attn",
25
+ "c_fc",
26
+ "c_attn"
27
  ],
28
  "task_type": "CAUSAL_LM",
29
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1f148bdf323d834952224e2ae4b017348008fc4c11c8fa07d046a363b9c3ad9f
3
  size 22241240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4aa5c58bfd5a52bab356a7c562fddb9d55aff986990047dfb66806b039f01ec3
3
  size 22241240
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": 0,
5
+ "transformers_version": "4.46.2"
6
+ }
runs/Nov19_13-49-20_localhost/events.out.tfevents.1732004362.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ddcc82a3767ffcfcebea0b73be76f7569da96b0fd850216e28054df9137d039
3
+ size 5400
runs/Nov19_13-52-01_localhost/events.out.tfevents.1732004522.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b08464490734767d7ad0a624e90d1c96850466c0d5c6ab8ee2b9dd4cc9d7bc07
3
+ size 5400
runs/Nov19_14-03-23_localhost/events.out.tfevents.1732005204.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d37887c67dcf2e336ffd7287f4845f8c328816f1c1040a81f4ba5b523857e029
3
+ size 5400
runs/Nov19_14-13-30_localhost/events.out.tfevents.1732005811.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c3dcdc6287506eaf736213323da1c5f4303db5e2421f4795db654548bb263b2
3
+ size 17714
runs/Nov21_13-53-49_localhost/events.out.tfevents.1732177431.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6e421491fe4a657ad225f5c6f3bfb03ac2e451a2813860ed195e089853b66cf6
3
+ size 5405
runs/Nov21_16-15-04_localhost/events.out.tfevents.1732185906.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:92814aaf00a38c012bc7a40a7d5350a0d83f628119b57676b7b38435481d7907
3
+ size 8758
runs/Nov21_17-19-19_localhost/events.out.tfevents.1732189761.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac07efe0c79f07c01b947d3b1523b165d857c4427f92e2c06208703130ac86ab
3
+ size 7312
runs/Nov21_17-23-58_localhost/events.out.tfevents.1732190040.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:318969b3f06cc6d724d1085c27dd9795fabc82e0599dc4ae46448f01b3e6ccd8
3
+ size 7312
runs/Nov21_18-47-25_localhost/events.out.tfevents.1732195047.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39d8915a487f2dd4d74faa4a599e1d152f70597deec10156c225d845dc9f5075
3
+ size 6348
runs/Nov21_19-13-55_localhost/events.out.tfevents.1732196637.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9f212c6546bc67bf958406bbf8616d79425eb62ad9ceb4792727fcf473fe006
3
+ size 10558
runs/Nov21_20-47-21_localhost/events.out.tfevents.1732202243.localhost ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:25f4f53257444d95d6014be2421b8999b4153bdae95d35c10c00c9e68bf2a68d
3
+ size 10205
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:aa42ca44218fc8c801de5af36377d87895d26ab0eb0d7aab7ba93fb9d218be55
3
  size 5304
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35a43dbd2422827738d27e88f16c12c5c64b18e213fdd77b162b4b2fbdd0a49f
3
  size 5304