littleworth/esm2_t33_650M_UR50D_pepmlm_lora_adapter_merged

Model Card for Model ID

This model builds upon PepMLM, aimed at generating peptides from receptor sequences. It incorporates the ESM model framework from HuggingFace for its core architecture. The key enhancement in this model is the adoption of LoRA for training, distinguishing it from its predecessor.

Usage:

from transformers import AutoTokenizer, AutoModelForMaskedLM
from peft import PeftConfig
import torch
model_name = "littleworth/esm2_t33_650M_UR50D_pepmlm_lora_adapter_merged"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name).to(device)

Packages version:

{'transformers': '4.36.0', 'peft': '0.9.0', 'torch': '2.0.0'}

Training summary:

{
  "train/loss": 1.5091,
  "train/grad_norm": 3.6427412033081055,
  "train/learning_rate": 6.773224309612687e-7,
  "train/epoch": 5,
  "train/global_step": 6395,
  "_timestamp": 1709229361.5373268,
  "_runtime": 25556.57973074913,
  "_step": 639,
  "train/train_runtime": 25557.6176,
  "train/train_samples_per_second": 4.003,
  "train/train_steps_per_second": 0.25,
  "train/total_flos": 220903283526564960,
  "train/train_loss": 1.8436848362317955,
  "_wandb": {
    "runtime": 25556
  }
}