Hugging Face's logo

JingyaoLi
/
MoTCoder-15B-v1.0

Text Generation
Transformers
Safetensors
gpt_bigcode
code
text-generation-inference
Inference Endpoints
Model card Files Community
JingyaoLi commited on
Commit
8d9e7c8
•
1 Parent(s): 5b09e3b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +15 -42
README.md CHANGED
@@ -26,49 +26,22 @@ Our investigations reveal that, through the cultivation and utilization of sub-m
26
  ![Performance on APPS](./impression.png)
27
 
28
  **Performance on APPS**
29
- | Model | Size | Pass@ | Introductory | Interview | Competition | All |
30
- |---------------------------|------|-------|--------------|-----------|-------------|------|
31
- | **GPT-Neo** | 2.7B | 1 | 3.90 | 0.57 | 0.00 | 1.12 |
32
- | | | 5 | 5.50 | 0.80 | 0.00 | 1.58 |
33
- | **Codex** | 12B | 1 | 4.14 | 0.14 | 0.02 | 0.92 |
34
- | | | 5 | 9.65 | 0.51 | 0.09 | 2.25 |
35
- | | | 1000 | 25.02 | 3.70 | 3.23 | 7.87 |
36
- | **AlphaCode** | 1B | 1000 | 17.67 | 5.24 | 7.06 | 8.09 |
37
- | **AlphaCode (Filtered 1k)**| | 5 | 14.36 | 5.63 | 4.58 | 7.17 |
38
- | **AlphaCode (Filtered 10k)**| | 5 | 18.18 | 8.21 | 6.65 | 9.89 |
39
- | **AlphaCode (Filtered 50k)**| | 5 | 20.36 | 9.66 | 7.75 | 11.42 |
40
- | **StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
41
- | **WizardCoder** | 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
42
- | **CodeLlama** | 7B | 5 | 10.76 | 2.01 | 0.77 | 3.51 |
43
- | | | 10 | 15.59 | 3.12 | 1.41 | 5.27 |
44
- | | | 100 | 33.52 | 9.40 | 7.13 | 13.77|
45
- | | 13B | 5 | 23.74 | 5.63 | 2.05 | 8.54 |
46
- | | | 10 | 30.19 | 8.12 | 3.35 | 11.58|
47
- | | | 100 | 48.99 | 18.40 | 11.98 | 23.23|
48
- | | 34B | 5 | 32.81 | 8.75 | 2.88 | 12.39|
49
- | | | 10 | 38.97 | 12.16 | 4.69 | 16.03|
50
- | | | 100 | 56.32 | 24.31 | 15.39 | 28.93|
51
- | **CodeLlama-Python** | 7B | 5 | 12.72 | 4.18 | 1.31 | 5.31 |
52
- | | | 10 | 18.50 | 6.25 | 2.24 | 7.90 |
53
- | | | 100 | 38.26 | 14.94 | 9.12 | 18.44|
54
- | | 13B | 5 | 26.33 | 7.06 | 2.79 | 10.06|
55
- | | | 10 | 32.77 | 10.03 | 4.33 | 13.44|
56
- | | | 100 | 51.60 | 21.46 | 14.60 | 26.12 |
57
- | | 34B | 5 | 28.94 | 7.80 | 3.45 | 11.16 |
58
- | | | 10 | 35.91 | 11.12 | 5.53 | 14.96 |
59
- | | | 100 | 54.92 | 23.90 | 16.81 | 28.69 |
60
- | **CodeLlama-Instruct** | 7B | 5 | 12.85 | 2.07 | 1.13 | 4.04 |
61
- | | | 10 | 17.86 | 3.12 | 1.95 | 5.83 |
62
- | | | 100 | 35.37 | 9.44 | 8.45 | 14.43 |
63
- | | 13B | 5 | 24.01 | 6.93 | 2.39 | 9.44 |
64
- | | | 10 | 30.27 | 9.58 | 3.83 | 12.57 |
65
- | | | 100 | 48.73 | 19.55 | 13.12 | 24.10 |
66
- | | 34B | 5 | 31.56 | 7.86 | 3.21 | 11.67 |
67
- | | | 10 | 37.80 | 11.08 | 5.12 | 15.23 |
68
- | | | 100 | 55.72 | 22.80 | 16.38 | 28.10 |
69
  | **MoTCoder** | 15B | 1 | **33.80** | **19.70** | **11.09** | **20.80** |
70
- | **code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
71
- | **GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
 
72
 
73
  **Performance on CodeContests**
74
  | Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |
 
26
  ![Performance on APPS](./impression.png)
27
 
28
  **Performance on APPS**
29
+ | Model | Size | Pass@ | Introductory | Interview | Competition | All |
30
+ |------------|-------|-------|--------------|-----------|-------------|-------|
31
+ | **CodeT5** | 770M | 1 | 6.60 | 1.03 | 0.30 | 2.00 |
32
+ | **GPT-Neo** | 2.7B | 1 | 14.68 | 9.85 | 6.54 | 10.15 |
33
+ | | | 5 | 19.89 | 13.19 | 9.90 | 13.87 |
34
+ | **GPT-2** | 0.1B | 1 | 5.64 | 6.93 | 4.37 | 6.16 |
35
+ | | | 5 | 13.81 | 10.97 | 7.03 | 10.75 |
36
+ | | 1.5B | 1 | 7.40 | 9.11 | 5.05 | 7.96 |
37
+ | | | 5 | 16.86 | 13.84 | 9.01 | 13.48 |
38
+ | **GPT-3** | 175B | 1 | 0.57 | 0.65 | 0.21 | 0.55 |
39
+ | **StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
40
+ | **WizardCoder**| 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  | **MoTCoder** | 15B | 1 | **33.80** | **19.70** | **11.09** | **20.80** |
42
+ | **text-davinci-002** | - | 1 | - | - | - | 7.48 |
43
+ | **code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
44
+ | **GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
45
 
46
  **Performance on CodeContests**
47
  | Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |