Bajiyo's picture
Upload tokenizer
c826bd5 verified
raw
history blame contribute delete
252 Bytes
{
"</s>": 64001,
"<2as>": 64002,
"<2bn>": 64003,
"<2en>": 64004,
"<2gu>": 64005,
"<2hi>": 64006,
"<2kn>": 64007,
"<2ml>": 64008,
"<2mr>": 64009,
"<2or>": 64010,
"<2pa>": 64011,
"<2ta>": 64012,
"<2te>": 64013,
"<s>": 64000
}