5fa1a76
1
2
3
There are two types of tokenizers you can use with 🤗 Transformers: [PreTrainedTokenizer]: a Python implementation of a tokenizer.
