id2pg_pattern2_ja_b8e15_test / added_tokens.json
takumi12's picture
Upload tokenizer
3acff90
raw
history blame
126 Bytes
{
"</s>": 2,
"<s>": 1,
"<unk>": 0,
"▁<EOT>": 32010,
"▁<MID>": 32009,
"▁<PRE>": 32007,
"▁<SUF>": 32008
}