Edit model card

NovelAI Tokenizer v1

This repository is exactly the same as NovelAI/nerdstash-tokenizer-v1, but the config has been changed to address the following points (the sp model itself is not changed).

  • Load as T5Tokenizer
  • Enable to decode digits (In the original, digits are registered as additional_special_tokens, so if skip_special_tokens=True when decoding, the digits are also skipped.)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mkshing/novelai-tokenizer-v1", use_fast=False)

text = "1+1=3"
tokenizer.decode(tokenizer.encode(text), skip_special_tokens=True)
# '1+1=3'
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .