Probability tensor contains either `inf`, `nan` or element < 0
I am trying to make inference with this code:
from transformers import GPTNeoXForCausalLM, AutoTokenizertokenizer = AutoTokenizer.from_pretrained("OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5")model = GPTNeoXForCausalLM.from_pretrained("OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5",device_map="auto", load_in_8bit=True)message = "<|prompter|>What is a meme, and what's the history behind this word?<|endoftext|><|assistant|>"inputs = tokenizer(message, return_tensors="pt").to(model.device)tokens = model.generate(**inputs, max_new_tokens=1000, do_sample=True, temperature=0.8)tokenizer.decode(tokens[0])
and it's returning this error all the time:RuntimeError: probability tensor contains either inf, nan or element < 0
Does anyone know how to solve it, my tensors have no 'nan' values or 'inf' I have checked them.
I was also having this issue. I found that if I set do_sample=False it would run, but do_sample=True would give the error you're getting. I need to investigate more to see what happened (not sure if I changed something else in the code or if one of the libraries was updated) as ideally, I want to have the option of using do_sample.