SQLGenerator / tokenizer.json
Kasivs's picture
Create tokenizer.json
4387ce7 verified
raw
history blame contribute delete
155 Bytes
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.save_pretrained("path_to_save_tokenizer")