Spaces:

xu-song
/

tokenizer-arena

Running

add more tokenizer

5425d5d 7 months ago

151 Bytes



	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b", trust_remote_code=True)

	tokenizer.comments = ""