Model Card for deepseek-coder-6.7b-vulnerability-detection

Fine-tuned version of deepseek-coder-6.7b-instruct aiming to improve vulnerability detection in solidity smart contracts and provide informative explanations on what the vulnerabilities are, and how to solve them.

Model Details

Model Description

Given the following prompt below:

Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.

Output only the description of the vulnerability and the attacking vector. No additional information is needed.

If there is no vulnerability output "There is no vulnearbility".

Codeblocks:
{}

When 1 or more codeblocks are provided to the model using this prompt, the model will output:

1. Wether there is a vulnerability or not.
2. What the vulnerability is.
3. How an attacker would take advantage of the detected vulnerability.

Afterwards, the above output can be chained to produce a solution - the context has the code, the vulnerability and the attacking vector so deducing a solution becomes a more straight-forward task. Additionally, the same fine-tuned model can be used for the solution recommendation as the fine-tuning is low-rank (LoRA) and a lot of the model ability is preserved.

• Developed by: [Kristian Apostolov]
• Shared by: [Kristian Apostolov]
• Model type: [Decoder]
• Language(s) (NLP): [English]
• License: [MIT]
• Finetuned from model: [deepseek-ai/deepseek-coder-6.7b-instruct]

Model Sources [optional]

• Repository: [https://huggingface.co/msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability-detection]

Uses

Provide code from a smart contract for a preliminary audit.

Direct Use

[More Information Needed]

Out-of-Scope Use

Malicious entity could detect 0-day vulnerability and take advantage of it.

Bias, Risks, and Limitations

The training data could be improved. Audits sometimes describe vulnerabilities which are not necessarily contained in the code itself, but are a part of a larger context.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model.

model_name = 'msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability'
tokenizer = AutoTokenizer.from_pretrained( # For some reason the tokenizer didn't save properly
    "deepseek-ai/deepseek-coder-6.7b-instruct", 
    trust_remote_code=True, 
    force_download=True,
)

prompt = \
"""
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.

Output only the description of the vulnerability and the attacking vector. No additional information is needed.

If there is no vulnerability output "There is no vulnearbility".

Codeblocks:
{}

"""

codeblocks = "Your code here"

messages = [
    { 'role': 'user', 'content': prompt.format(codeblocks) }
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, top_k=25, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
description = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True) 

print(description)

Training Details

Training Data

https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons

Training Procedure

lora_config = LoraConfig( r=16, # rank lora_alpha=32, # scaling factor target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_dropout=0.05, # dropout rate for LoRA layers )

TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, num_train_epochs = 1, learning_rate = 2e-4, fp16 = True, logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", )

Training Hyperparameters

• Training regime: fp16 mixed precision

Evaluation

Testing Data, Factors & Metrics

Testing Data

https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons

Framework versions

• PEFT 0.11.1