Match official tokenizer_config.json. Changes were made after this repo was created.
#8
by
Tostino
- opened
No description provided.
I believe you also have to fix the special token map to match.
"eos_token": "<|end_of_text|>"
"eos_token": "<|eot_id|>"
teknium
changed pull request status to
merged
merged