falcon-11B-vlm / README.md
rcojocaru's picture
Update README.md
be99415 verified
metadata
language:
  - en
inference: false
license: unknown

πŸš€ Falcon2-11B-vlm

Falcon2-11B-vlm is an 11B parameters causal decoder-only model built by TII and trained on over 5,000B tokens of RefinedWeb enhanced with curated corpora. To bring vision capabilities, we integrate the pretrained CLIP ViT-L/14 vision encoder with our Falcon2-11B chat-finetuned model and train with image-text data. For enhancing the VLM's perception of fine-grained details w.r.t small objects in images, we employ a dynamic encoding mechanism at high-resolution for image inputs. The model is made available under the TII Falcon License 2.0, the permissive Apache 2.0-based software license which includes an acceptable use policy that promotes the responsible use of AI.

Paper coming soon 😊.

πŸ€— To get started with Falcon-vlm (inference, finetuning, quantization, etc.), we recommend reading this great blogpost from HF!

from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from PIL import Image
import requests
import torch

processor = LlavaNextProcessor.from_pretrained("tiiuae/falcon-11B-vlm", tokenizer_class='PreTrainedTokenizerFast')
model = LlavaNextForConditionalGeneration.from_pretrained("tiiuae/falcon-11B-vlm", torch_dtype=torch.bfloat16)


url = "http://images.cocodataset.org/val2017/000000039769.jpg"
cats_image = Image.open(requests.get(url, stream=True).raw)
instruction = 'Write a long paragraph about this picture.'

prompt = f"""User:<image>\n{instruction} Falcon:"""
inputs = processor(prompt, images=cats_image, return_tensors="pt", padding=True).to('cuda:0')

model.to('cuda:0')
output = model.generate(**inputs, max_new_tokens=256)


prompt_length = inputs['input_ids'].shape[1]
generated_captions = processor.decode(output[0], skip_special_tokens=True).strip()

print(generated_captions)

πŸ’₯ Falcon VLMs require PyTorch 2.0 for use with transformers!

For fast inference with Falcon, check-out Text Generation Inference! Read more in this blogpost.

Model Card for Falcon2-11B-VLM

Model Details

Model Description

Model Source

  • Paper: coming soon.

Uses

Direct Use

Research on General large vison language models.

Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.

Bias, Risks, and Limitations

Falcon2-11B is trained mostly on English, but also German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish. It will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

Training Details

The training is done in two stages: pretraining and finetuning. In both stages, the visual encoder weights are kept frozen. In the pretraining stage, the LLM is kept frozen and only the multimodal projector is trained on 558K image-caption pairs. This enables the multimodal projector to learn a mapping from visual to text embedding space. During finetuning, both the projector and LLM weights are trained on a corpus of 1.2M image-text instruction data from public datasets, which also includes multi-round conversations. Falcon2-11B-VLM was trained on 16 A100 80GB GPUs with ZeRO and Flash-Attention 2.

The data was tokenized with the Falcon-7B/11B tokenizer.

Training Hyperparameters

Hyperparameter Value
Precision bfloat16
Optimizer AdamW
Max learning rate 2e-5
Weight decay 0
Batch size 256

Evaluation

Model MME GQA SQA POPE VQAv2 TextVQA MM-Bench SEED-IMG
Falcon2-11B VLM 1589/343 64.5 74.9 88.4 82.1 66.7 72.0 72.3

Citation

Paper coming soon 😊.

License

Falcon2-11B is licenced under TII Falcon License 2.0, the permissive Apache 2.0-based software license which includes an acceptable use policy that promotes the responsible use of AI.

Contact

falconllm@tii.ae