Salesforce/codegen-2B-mono · ValueError: Tokenizer class CodeGenTokenizer does not exist or is not currently imported.

Jul 1, 2022

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-2B-mono")
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen-2B-mono")

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

ValueError: Tokenizer class CodeGenTokenizer does not exist or is not currently imported.

rooa

Salesforce org Jul 1, 2022

@wepin Could you let me know the transformers version? Our model is not included in a release yet, so you'd have to install from the repo, i.e.

pip install git+https://github.com/huggingface/transfomers.git

rooa changed discussion status to closed Jul 4, 2022