Tokenizer is not working with BartTokenizer
Hello,
Could you please provide more information on how to use the model and tokenizer?
The config file has "tokenizer_class": "SciBartTokenizer" but SciBartTokenizer is not a Huggingface Tokenizer class. Therefore, when I use AutoTokenizer.from_pretrained("scibart-base") or BartTokenizer.from_pretrained("scibart-base"), I get an error.
Could you please provide the huggingface code to load the tokenizer and the model?
Thank you.
Hello,
Thank you for answering.
I have followed the instructions on the github, I have now transformers 4.30.0.dev0 from your branch.
However, when I try to load the tokenizer using AutoTokenizer, I still get the error "ValueError: Tokenizer class SciBartTokenizer does not exist or is not currently imported". This error happens regardless of the model I use (scibart-base or scibart-large).
Thank you,
Maël
Hi,
Thanks for reporting this issue. However, we are unable to reproduce this error. We tried creating a new conda testing environment and installing transformers cloned from git clone https://github.com/xiaowu0162/transformers.git -b scibart-integration. Then both the AutoTokenizer and SciBartTokenizer work normally with SciBART. If you use an environment, can you check if the environment is activated?
Best,
Di
Hi,
Thank you, I indeed use an environment and yes I verified if the environment was activated and also if the new version of transformers was correctly installed. I used the same command line as you to clone the needed version of transformers.
I tried with 3 environments: 1 older, the one I am currently working on and a brand new environment.
I had the bug with the last two. However, with the first one, it went smoothly so it seems that the problem is on my side.
To look at the difference between the two environments, I did a pip list.
The one that works has this for transformers transformers 4.30.0.dev0 /home/<my_username>/transformers/src
For the two other environments that bring the error, I have this transformers 4.30.0.dev0 /home/<my_username>/transformers
I think it may have something to do with the versions of pip there is on my latest environments.
I will close this issue as the problem definitely comes from my environments.
Thank you for your time.