File size: 2,022 Bytes
76ad19a eff144c 76ad19a 533ef56 76ad19a 7c25f52 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: mit
datasets:
- nielsr/docvqa_1200_examples_donut
language:
- en
library_name: transformers
pipeline_tag: visual-question-answering
---
### IDEFICS2-OCR
Finetuned of Idefics2-8b with fp16 weight update on nielsr/docvqa_1200_examples_donut dataset for document VQA pairs.
## Usage
```Python
from transformers import BitsAndBytesConfig, AutoModelForVision2Seq, AutoProcessor
from transformers.image_utils import load_image
processor = AutoProcessor.from_pretrained("smishr-18/Idefics2-OCR", do_image_splitting=False)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16
)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = AutoModelForVision2Seq.from_pretrained(
"smishr-18/Idefics2-OCR",
quantization_config=bnb_config,
device_map=device,
low_cpu_mem_usage=True
)
image = load_image("https://images.pokemontcg.io/pl1/1_hires.png")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Explain."},
{"type": "image"},
{"type": "text", "text": "What is the reflex energy in the image?"}
]
}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[text.strip()], images=[image4], return_tensors="pt", padding=True)
inputs = {k: v.to(device) for k, v in inputs.items()}
# Generate texts
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_texts)
# The reflex energy in the image is 70.
```
## Limitations
The model was finetuned on limited T4 GPU and could be fintuned with more adapters on
devices with ```torch.cuda.get_device_capability()[0] >= 8``` or Ampere GPUs.
- **Developed by:** Shubh Mishra, Aug 2024
- **Model Type:** VLM
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model:** HuggingFaceM4/idefics2-8b |