tokenization_xgen.py not being loaded automagically
#3
by
gwagner
- opened
The current configuration is not automagically loading XgenTokenizer
Tokenizer class XgenTokenizer does not exist or is not currently imported.
This is a custom tokenizer, defined in this repository. Please add trust_remote_code=True when calling AutoTokenizer, as in:
tokenizer = AutoTokenizer.from_pretrained("Salesforce/xgen-7b-4k-base", trust_remote_code=True)
This worked. Thanks!
gwagner
changed discussion status to
closed