ValueError: Need either a `state_dict` or a `save_folder` containing offloaded weights.
I am using Mac M1.
I am using code:
"from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")
input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))"
The error:
File ~/miniforge3/envs/tf/lib/python3.8/site-packages/accelerate/utils/offload.py:154, in OffloadedWeightsLoader.init(self, state_dict, save_folder, index, device)
146 def init(
147 self,
148 state_dict: Dict[str, torch.Tensor] = None,
(...)
151 device=None,
152 ):
153 if state_dict is None and save_folder is None:
--> 154 raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.")
156 self.state_dict = {} if state_dict is None else state_dict
157 self.save_folder = save_folder
ValueError: Need either a state_dict or a save_folder containing offloaded weights.
Have the same issue.
Have the same issue but while loading Llama. Need immediate help.
cc @marcsun13
Have the same issue but while loading Llama. Need immediate help.
Same issue while loading llama.
Could you provide a minimal reproducer @liut89 and @UdheepSai . I tried the script on T5 xxl on mac and it works fine with the offload.