Something is wrong with gpt4 tokenizer
#7
by
mahnerak
- opened
Hi there. Which tokenizer are you using? It seems to be working fine on my side. :)
We did make an update to the tokenizer.json over 2 months ago (see commit), which may have affected this, so you may need to reset the cache if you still have the old version cached. For example, if in chrome:
- Open dev tools (F12)
- Go to Application Tab
- Click "Cache storage" dropdown
- Right click "transformers-cache" and click "Delete"
Let me know if that helps!
Thanks
Even after clearing everything (from application tab) the problem persisted :/
But I can confirm that with other devices and Incognito everything works well.
mahnerak
changed discussion status to
closed