Model Card for Starcoder2-3B Fine-Tuned with QLORA on Solidity Dataset
Model Details
Model Description
This model has been fine-tuned using QLORA specifically for generating Solidity smart contracts from natural language instructions.
- Developed by: @yoniebans
- Model type: Transformer-based Causal Language Model
- Language(s) (NLP): English with programming language syntax (Solidity)
- License: [More Information Needed]
- Finetuned from model bigcode/starcoder2-3b
Model Sources [optional]
- Repository: [More Information Needed]
- Demo [optional]: [More Information Needed]
Uses
Direct Use
This model is designed to demonstrate how fine-tuning a base model such as starcoder2-3b using qlora with a targetted dataset can increase the model's ability to create Solidity code directly from natural language descriptions or instructions.
Out-of-Scope Use
This model is not intended for:
- Deployment in production systems without rigorous testing.
- Use in non-technical text generation or any context outside smart contract development.
Bias, Risks, and Limitations
The training data primarily consists of code from publicly sourced Solidity projects which may not encompass the full diversity of programming styles and techniques.
Recommendations
Users are advised to use this model as a starting point for development and not as a definitive solution. Generated code should always be reviewed by experienced developers to ensure security and functionality.
How to Get Started with the Model
Use the code below to get started with the model.
import sys
from peft import PeftModel
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
import torch
import accelerate
use_4bit = True
bnb_4bit_compute_dtype = "float32"
bnb_4bit_quant_type = "nf4"
use_double_nested_quant = True
compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
device_map = "auto"
max_memory = '45000MB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}
model = AutoModelForCausalLM.from_pretrained(
"bigcode/starcoder2-3b",
cache_dir=None,
device_map=device_map,
max_memory=max_memory,
quantization_config=BitsAndBytesConfig(
load_in_4bit=use_4bit,
llm_int8_threshold=6.0,
llm_int8_has_fp16_weight=False,
bnb_4bit_compute_dtype=compute_dtype,
bnb_4bit_use_double_quant=use_double_nested_quant,
bnb_4bit_quant_type=bnb_4bit_quant_type
),
torch_dtype=torch.float32,
trust_remote_code=False
)
adapter_weights = "yoniebans/starcoder2-3b-qlora-solidity"
model = PeftModel.from_pretrained(model, adapter_weights)
tokenizer = AutoTokenizer.from_pretrained(
checkpoint,
cache_dir=None,
padding_side="right",
use_fast=False,
tokenizer_type=None, # Needed for HF name change
trust_remote_code=False,
use_auth_token=False,
)
input='Make a smart contract for a memecoin named 'LLMAI', adhering to the ERC20 standard. The contract should enforce a purchase limit where no individual wallet can acquire more than 1% of the total token supply, which is set at 10 billion tokens. This purchasing limit should be modifiable and can only be disabled by the contract owner at their discretion. Note that the interfaces for ERC20, Ownable, and any other dependencies should be assumed as already imported and do not need to be included in your code response.'
prompt = f"""### Instruction:
Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:
### Task:
{input}
### Solution:
"""
input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = model.generate(
input_ids=input_ids,
max_new_tokens=2048,
do_sample=True,
top_p=0.9,
temperature=0.001,
pad_token_id=1
)
output_text = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
output_text_without_prompt = output_text[len(prompt):]
file_path = './smart_contract.sol'
with open(file_path, 'w') as file:
file.write(output_text_without_prompt)
print(f"Output written to {file_path}")
Training Details
Training Data
The model was trained on a dataset consisting of pairs of natural language instructions and their corresponding Solidity code implementations. This dataset includes over 6000 input and outputs.
Training Procedure
Training Hyperparameters
- Number of train epochs: 30
- Double quant: true
- Quant type: nf4
- Bits: 4
- Lora r: 64
- Lora aplha: 16
- Lora dropout: 0.0
- Per device train batch size: 1
- Gradient accumulation steps: 1
- Max steps: 10000
- Weight decay: 0.0 #use lora dropout instead for regularization if needed
- Learning rate: 2e-4
- Max gradient normal: 0.3
- Gradient checkpointing: true
- FP16: false
- FP16 option level: O1
- BF16: false
- Optimizer: paged AdamW 32-bit
- Learning rate scheduler type: constant
- Warmup ratio: 0.03
Speeds, Sizes, Times [optional]
| Epoch | Grad Norm | Loss | Step |
|---|---|---|---|
| 0.03 | 0.513248 | 2.0948 | 10 |
| 0.59 | 0.976398 | 0.7952 | 200 |
| 1.18 | 0.291436 | 0.5081 | 400 |
| 1.78 | 0.197318 | 0.5007 | 600 |
| 2.37 | 0.143200 | 0.4189 | 800 |
| 2.96 | 0.153539 | 0.4314 | 1000 |
| 3.55 | 0.199708 | 0.4554 | 1200 |
| 4.15 | 0.258209 | 0.4856 | 1400 |
| 4.74 | 0.295185 | 0.5446 | 1600 |
| 5.33 | 0.157903 | 0.3209 | 1800 |
| 5.92 | 0.149965 | 0.3203 | 2000 |
| 6.52 | 0.166200 | 0.4046 | 2200 |
| 7.11 | 0.181177 | 0.2157 | 2400 |
| 7.70 | 0.156556 | 0.2937 | 2600 |
| 8.29 | 0.309143 | 0.2928 | 2800 |
| 8.89 | 0.211304 | 0.3414 | 3000 |
| 9.48 | 0.166119 | 0.2716 | 3200 |
| 10.07 | 0.265576 | 0.1727 | 3400 |
| 10.66 | 0.231398 | 0.2219 | 3600 |
| 11.26 | 0.183477 | 0.1706 | 3800 |
| 11.85 | 0.182451 | 0.1471 | 4000 |
| 12.44 | 0.272920 | 0.1793 | 4200 |
| 13.03 | 0.189667 | 0.1241 | 4400 |
| 13.62 | 0.136364 | 0.1278 | 4600 |
| 14.22 | 0.297066 | 0.1043 | 4800 |
| 14.81 | 0.213272 | 0.1760 | 5000 |
| 15.40 | 0.162718 | 0.1062 | 5200 |
| 15.99 | 0.229559 | 0.1012 | 5400 |
| 16.59 | 0.271037 | 0.1180 | 5600 |
| 17.18 | 0.239012 | 0.0871 | 5800 |
| 17.77 | 0.175727 | 0.0894 | 6000 |
| 18.36 | 0.151963 | 0.1154 | 6200 |
| 18.96 | 0.202392 | 0.1096 | 6400 |
| 19.55 | 0.202703 | 0.0764 | 6600 |
| 20.18 | 0.148534 | 0.0551 | 6800 |
| 20.77 | 0.151745 | 0.0599 | 7000 |
| 21.36 | 0.188332 | 0.0707 | 7200 |
| 21.95 | 0.190111 | 0.0987 | 7400 |
| 22.55 | 0.163640 | 0.0567 | 7600 |
| 23.14 | 0.209854 | 0.0534 | 7800 |
| 23.73 | 0.225155 | 0.0626 | 8000 |
| 24.32 | 0.172093 | 0.0389 | 8200 |
| 24.92 | 0.162895 | 0.0347 | 8400 |
| 25.52 | 0.187249 | 0.0592 | 8600 |
| 26.11 | 0.182813 | 0.0368 | 8800 |
| 26.70 | 0.243531 | 0.0427 | 9000 |
| 27.29 | 0.190624 | 0.0325 | 9200 |
| 27.89 | 0.189085 | 0.0311 | 9400 |
| 28.48 | 0.171129 | 0.0322 | 9600 |
| 29.07 | 0.138704 | 0.0353 | 9800 |
| 29.66 | 0.216177 | 0.0312 | 10000 |
Results
Summary
Initial evaluations show promising results in generating Solidity code, with further testing planned.
Environmental Impact
- Hardware Type: NVIDIA A40 GPU, 48GB VRAM, 48GB RAM, 9vCPU
- Hours used: 18
- Cloud Provider: runpod.io
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Citation [optional]
BibTeX:
@misc{lozhkov2024starcoder,
title={StarCoder 2 and The Stack v2: The Next Generation},
author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},
year={2024},
eprint={2402.19173},
archivePrefix={arXiv},
primaryClass={cs.SE}
}
APA:
Starcoder2-3B: BigCode Team. (2024). Starcoder2-3B [Software]. Available from https://huggingface.co/bigcode/starcoder2-3b
AlfredPros Smart Contracts Instructions: AlfredPros. (2023). Smart Contracts Instructions [Data set]. Available from https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions