Text only generation for Idefics3

#22
by FreekCool - opened

Hi everyone, is it possible to only pass text to Idefics3, so text to text generation? I am using approximately the same code as with Idefics2, which seems to be in line with Idefics3. In the following code I do not pass images to the prompt:

Create inputs

        messages = [
            {
                "role": "user",
                "content": [
                ]
            }
        ]

        format_prompt = prompt.format(truncated_text)
        messages[0]['content'].append({"type": "text", "text": format_prompt})

        idefics_prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
        inputs = processor(text=idefics_prompt, return_tensors="pt")
        inputs = {k: v.to(DEVICE) for k, v in inputs.items()}

However, this line "inputs = processor(text=idefics_prompt, return_tensors="pt")" gives me the following error:

ValueError: The number of images in the text [0] and images [] should be the same.

Does anyone know if there is a solution? I installed the newest Transformers version also from source but it does not work. Thanks in advance!

I have the same problem

It works by just tokenizing (using Idefics3 tokenizer) the idefics_prompt as the input.

It works by just tokenizing (using Idefics3 tokenizer) the idefics_prompt as the input.
@Guanzheng
Something like ?

inputs = tokenizer.apply_chat_template(
        messages,
        tokenize = True,
        add_generation_prompt = True, # False for fine-tuning
        return_tensors = "pt",).to(DEVICE)

Sign up or log in to comment